margaux Posted November 6, 2012 Share Posted November 6, 2012 I'm working on a site which reads in data from an xml feed and creates a d/b table. The feed is continually being updated so the table also has to be updated frequently. I think the feed will be accessed several times/hour to refresh the database. Is the better approach to 1. Drop the table and recreate from fresh everytime the feed comes in. This ensures that the database doesn't continually grow with items that have already been stored. However how will site visitors be affected if the table is being created while they're viewing? 2. Create the table once and update it subsequently. Check for each item's unique reference - if it exists then overwrite it, that way you pick up any changes to the item. This approach seems less attractive from a speed point of view. Also it means that items that are no longer on the feed will still be on the table and there will have to be some kind of manual reconciliation? Or is there another option? Thanks for your thoughts. Link to comment Share on other sites More sharing options...
Larry Posted November 7, 2012 Share Posted November 7, 2012 It really depends upon the data and how it's used. The idea of wiping out a database and repopulating it with each data request sounds very strange to me, though. Link to comment Share on other sites More sharing options...
Antonio Conte Posted November 7, 2012 Share Posted November 7, 2012 I would just always select the X latest rows and let it be with that. DBMS systems can handle huge sets of data, so I would not worry about that. If it should become a problem, run a clean up function once in a while instead. This can of course be automated by writing a cron job. 1 Link to comment Share on other sites More sharing options...
margaux Posted November 7, 2012 Author Share Posted November 7, 2012 I would just always select the X latest rows and let it be with that. DBMS systems can handle huge sets of data, so I would not worry about that. If it should become a problem, run a clean up function once in a while instead. That seems like a better approach but how does that cater for items that are no longer on the feed but will still be on the database and returned by search queries? Link to comment Share on other sites More sharing options...
Recommended Posts