DataList approach for large datasets without Counts

Aug 31, 2012 at 6:11 PM

How would I approach a scenario where I have a large OData source which does not support count options?

I can't use PagedDataListSource as I don't know the total number of entries in a dataset.

I've been looking at the IncrementalLoadingDataList, but there are no examples of this in the samples download.

Sep 1, 2012 at 6:16 AM
Edited Sep 1, 2012 at 6:19 AM

My question is on the Count function. Since my OData source has a max of 1000 per request and it doesn't support count, how would I get the total count. Let me know if I'm doing something stupid :)

This is my Customer Api (The problem with this part is that the GetCustomerCount() will only return a max of 1000 as that is the limit of the number I can pull down):

        #region Customer
        public async Task<int> GetCustomerCount()
        {
            Uri resourceUri = new Uri(_rootUri.ToString() + "Customer");
            SyndicationFeed feed = await GetClient().RetrieveFeedAsync(resourceUri);
            return feed.Items.Count();        
        }

        public async Task<CustomerPage> GetCustomers(int index)
        {
            string token = string.Format("?$top={0}&$skip={1}", 1, index);
            Uri resourceUri = new Uri(_rootUri.ToString() + "Customer" + token);

            XDocument responseDocument = XDocument.Parse("<customers />");

            SyndicationFeed feed = await GetClient().RetrieveFeedAsync(resourceUri);
            foreach (SyndicationItem item in feed.Items)
            {
                XElement ele = XElement.Parse(item.Content.Xml.FirstChild.GetXml());
                ele.DescendantsAndSelf().Attributes().Where(a => a.IsNamespaceDeclaration).Remove();
                var all = ele.DescendantsAndSelf();
                foreach (var el in all)
                    el.Name = el.Name.LocalName;
                responseDocument.Root.Add(ele);
            }            

            XElement rspElement = responseDocument.Root;

            XmlSerializer serializer = new XmlSerializer(typeof(CustomerPage));
            object result = serializer.Deserialize(rspElement.CreateReader());

            return (CustomerPage)result;
        }
        #endregion
This is my CustomerDataListSource:
    public class CustomerDataListSource : IDataListSource<Customer>
    {
        private CustApi _custApi;

        public CustomerDataListSource(CustApi custApi)
        {
            this._custApi = custApi;
        }

        public async Task<int> GetCountAsync()
        {
            return await _custApi.GetCustomerCount();
        }

        public async Task<Customer> GetItemAsync(int index)
        {
            CustomerPage customerPage = await _custApi.GetCustomers(index);
            Customer cust = customerPage.Customers.FirstOrDefault();
            if (cust != null)
                cust.Index = index;
            return cust;
        }

        public int IndexOf(Customer item)
        {
            if (item != null)
                return item.Index;

            return -1;
        }

        public IDisposable Subscribe(IUpdatableCollection collection)
        {
            throw new NotImplementedException();
        }
    }

Sep 1, 2012 at 8:58 PM

I changed my count function, anyone see a problem with that?

        public async Task<int> GetCustomerCount()
        {
            int currentIndex = 1000;

            while (true)
            {
                string token = string.Format("?$top={0}&$skip={1}", 1, currentIndex);
                Uri resourceUri = new Uri(_rootUri.ToString() + "Customer" + token);
                SyndicationFeed feed = await GetClient().RetrieveFeedAsync(resourceUri);
                if (feed.Items.Count == 0)
                {
                    currentIndex -= 1000;
                    token = string.Format("?$skip={0}", currentIndex);
                    resourceUri = new Uri(_rootUri.ToString() + "Customer" + token);
                    feed = await GetClient().RetrieveFeedAsync(resourceUri);
                    currentIndex += feed.Items.Count;
                    return currentIndex;
                }
                else
                {
                    currentIndex += 1000;
                }
            }
        }

Sep 1, 2012 at 11:35 PM
Edited Sep 2, 2012 at 12:23 AM

I'm getting an exception when using the IncrementalDataList.

HRESULT: 0x80072EE2

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Data.CustApi.<GetItems>d__12`1.MoveNext() in c:\Visual Studio 2012\Projects\Data\CustApi.cs:line 154

The strange part is that the application runs for about 20-40 seconds and I get my data displayed in my GridView, but then the error pops up without moving around in the app or anything. I only have 68 items in my data list. The displaying of the grid shows one item right away, but then it takes a while to show the rest.

Coordinator
Sep 3, 2012 at 8:47 PM
Edited Sep 3, 2012 at 8:47 PM

Sorry it took me a while to get back to you on this - this one is a little trickier and I've been trying to get through everyone's questions.

Regarding the performance issues you are seeing, are you doing any caching here? I think the main issue is that the call to GetCustomerCount() and every single GetCustomers(...) call will make a web request so there is a lot of delays here. A much better approach is to retrieve a page of items at a time (e.g. '$top=20'). Whilst each web request will be larger (and you might download items that are not required) overall the performance will be much better. The second issue is that the Okra DataList implementations assume that it is the responsibility of the data list source to provide caching of the results - since you do not perform any caching then you may find that the same web requests are being called multiple times.

The good news is that the Okra App Framework includes a PagedDataListSource<T> class that handles such situations. There are three methods to implement FetchCountAsync(), FetchPageSizeAsync() and FetchPageAsync(). Note that you can return other data in each call, so the FetchCountAsync() method can return the total count but also the first page of data if available in the same call (lots of web APIs you retrieve the count as part of the first page of data). The PagedDataListSource<T> class also handles caching of the data so each request should only be made once. Check out this code from a previous sample (I need to release an updated version actually).

Something like the following (NB: I'm typing from memory so sorry for any typos!),

    public class CustomerDataListSource : PagedDataListSource<Customer>
    {
        ...

        // *** Overriden Base Methods ***

        protected async override Task<DataListPageResult<Customer>> FetchCountAsync()
        {
            int count = GetCustomerCount();
            return new DataListPageResult<Customer>(count , 20, null, null);
        }

        protected override Task<DataListPageResult<Customer>> FetchPageSizeAsync()
        {
            return Task.FromResult(new DataListPageResult<Customer>(null, 20, null, null););
        }

        protected async override Task<DataListPageResult<Customer>> FetchPageAsync(int pageNumber)
        {
            // Retrieve a page at a time (params are skip & top)
            Customer[] customerPage = GetCustomers((pageNumber - 1) * 20, 20);

            return new DataListPageResult<Customer>(null, 20, pageNumber, customerPage);
        }
    }

The bad news is that the Okra data framework doesn't support the case where you don't know the number of items so you will have to use your modified GetCustomerCount() method for this (although the base class would cache the count after the first call). The alternative would be to implement ISupportIncrementalLoading yourself - Mike Taulty has a good set of code here that you may find interesting.

Hope this helps,

    Andy

Sep 4, 2012 at 3:55 AM

Awesome Andy!

Works much better. Errors went away.

I'll start building in some caching next.

Thank for all your feedback.

Coordinator
Sep 4, 2012 at 8:56 AM

Great to hear that the alternative implementation fixed some of the issues.

Regarding caching - The PagedDataListSource<T> class holds an in memory cache of the items retrieved so you don't need to worry about that part. If you make it a singleton (e.g. by importing it using MEF and marking it as [Shared]) then you can also share it between different parts of your application or multiple views. It should then avoid having to requery the data a second time. Note that this is not persisted to disk so is constrained to the lifetime of the app, and will have to re-load the data if terminated.

Andy

Sep 11, 2012 at 1:10 PM
Edited Sep 11, 2012 at 2:02 PM

Andy,

I've tried getting the [Shared] to work for a DataSource from a different ViewModel, but can't seem to figure it out.

I use the [ImportingConstructor] on the viewmodel with an interface to my DataSource, but it still calls the constructor of the DataSource. 

I read the post : http://okra.codeplex.com/discussions/358012

but I can't seem to get a valid CompositionContextFactory back: 

[Import]

public ExportFactory<CompositionContext> CompositionContextFactory { get; set; } 



Do you have a simple example for sharing the DataSource?

Coordinator
Sep 11, 2012 at 8:31 PM

To share a singleton instance with MEF you should firstly export the data-source as shown below. Note that you do not use the argument on [Shared] as discussed in the post you linked to (this is for exactly the situation you do not want, the data source to be shared only within each page),

[Export(typeof(IMyDataSource))]
[Shared]
public class MyDataSource : IMyDataSource
{
    ...
}

Then you can import either via an importing constructor,

[ViewModelExport("MyPageName")]
public class MyViewModel
{
    [ImportingConstructor]
    public MyViewModel(IMyDataSource dataSource)
    {
        ...
    }
}

Or via an import property,

[ViewModelExport("MyPageName")]
public class MyViewModel
{
    [Import]
    public IMyDataSource DataSource {get;set;}

    ...
}

Note that you generally shouldn't need to use the ExportFactory<T> or the CompositionContext. The post you linked to was describing a more advanced usage than is generally needed and most of the time a simple Export/Import should work.

Regarding samples I'm aiming to put together an updated version of the Flickr browser sample that I released with older versions of the framework. This will have a shared data source between multiple pages so might help.

Andy

Sep 11, 2012 at 8:46 PM

Thanks Andy,

I tried this, but when using the [ImportConstructor] it re-initializes the DataSource class, if I use the [Import] the Datasource is null.

I'm basically just doing what you do in the FlickerApp, but I must be missing something. That's why I thought maybe it was more advanced.

John

Coordinator
Sep 11, 2012 at 8:55 PM

That's odd then. Just to check the basics - are you saying that you re-initialize the DataSource class with multiple view-models within the same application instance, or on that it re-initializes the DataSource when you start a new instance of the app? MEF will only constrain singletons whilst the app is in memory. Also if you are using the [Import] attribute then the property may need to be public with a public getter and setter (I think).

Otherwise I'm stumped - MEF normally handles the composition or gives an error so I suspect it is something simple though.

Sorry I can't be more help without seeing some code (maybe a stub would help?),

Andy

Sep 11, 2012 at 9:05 PM
Edited Sep 11, 2012 at 9:10 PM

I am stumped too

It's all within the same app. All I'm trying to do is to simulate the Standard ItemDetailView where you can browse thru the Items one by one, so I wanted to use the same datasource when a user clicks the item in the group view.

The code is posted below. when I click on a customer in the listview and it calls my constructor on the DetailViewModel it calls the Constructor of the DataSource even though the ListView just used the same DataSource.

    [Export(typeof(ICustomerDataSource))]
    [Shared]
    public class CustomerDataSource : ICustomerDataSource
    {
        private Api _api;
        private IList<Customer> _customers;
        private CustomerDataListSource _dataListSource;

        public CustomerDataSource()
        {
            _api = new Api();
            _dataListSource = new CustomerDataListSource(_api);
        }

        public IList<Customer> GetCustomers()
        {
            // Create a single instance of the customer list
            // This can then be shared between multiple view models
            if (_customers == null)
            {
                _customers = (IList<Customer>)new VirtualizingDataList<Customer>(_dataListSource);
            }

            return _customers;
        }
    }

    public class CustomerDetailViewModel : BindableBase, IActivatable<Customer, string>
    {
        private Customer _selectedCustomer;
        private IList<Customer> _customers;

        public CustomerDetailViewModel()
        {
            this.GoBackCommand = new DelegateCommand(GoBack);

            CustomerDataSource dataSource = new CustomerDataSource();
            this.Customers = dataSource.GetCustomers();
        }

        [ImportingConstructor]
        public CustomerDetailViewModel(ICustomerDataSource dataSource)
        {
            this.GoBackCommand = new DelegateCommand(GoBack);

            this.Customers = dataSource.GetCustomers();
        }

        [Import]
        public INavigationManager NavigationManager { get; set; }

        public ICommand GoBackCommand { get; private set; }

        public bool CanGoBack
        {
            get { return NavigationManager.CanGoBack; }
        }

        public void GoBack()
        {
            NavigationManager.GoBack();
        }

        public void Activate(Customer arguments, string state)
        {
            if (arguments == null)
                this._selectedCustomer = this.Customers.FirstOrDefault();
            else
                this._selectedCustomer = arguments;
        }

        public string SaveState()
        {
            return null;
        }

        public Customer SelectedCustomer
        {
            get 
            {
                if (this._selectedCustomer == null)
                {
                    this.SelectedCustomer = this.Customers.FirstOrDefault();
                }

                return this._selectedCustomer; 
            }
            set
            {
                if (_selectedCustomer != value)
                {
                    _selectedCustomer = value;
                    SetProperty(ref _selectedCustomer, value);
                }
            }
        }

        public IList<Customer> Customers
        {
            get
            {
                return _customers;
            }
            set
            {
                if (_customers != value)
                {
                    _customers = value;
                    SetProperty(ref _customers, value);
                }
            }
        }
    }

 

 

 

Coordinator
Sep 11, 2012 at 9:18 PM

Looks fine to me - maybe comment out the parameterless CustomerDetailViewModel() constructor and see if that helps. Since the view-model should be created by MEF it should always call the [ImportingConstructor] so the other constructor is not really needed. Either that or put a break-point in each constructor to confirm that the parameterless one is never called (since this calls the CustomerDataSource constructor directly MEF can't do anything to enforce a singleton) As an aside, I do normally include an empty protected parameterless constructor so that I can create a design-time view-model but shouldn't be needed at run-time.

The MEF website and forums http://mef.codeplex.com is also a good place to search for solutions.

Andy

Sep 11, 2012 at 9:22 PM
Edited Sep 12, 2012 at 4:41 AM

I'll try that out. I understand now it's all standard MEF. I'm new to MEF, so I wasn't sure if this was your framework or MEF causing this.

Thanks again!

Edit:

I think I figured it out : http://mef.codeplex.com/discussions/395216

Coordinator
Sep 12, 2012 at 11:19 AM

Good to hear that you got it sorted - MEF takes a little getting used to but once you have your head around it can be really powerful.

Andy