Memory issue with FDO IInsert and workaround

classic Classic list List threaded Threaded
4 messages Options
Ray
Reply | Threaded
Open this post in threaded view
|

Memory issue with FDO IInsert and workaround

Ray

I am importing a very large data set from ArcGIS to Autocad MAP 3d using FDO. However, I am hit by "out of memory" issue with FDO IInsert to insert data to a feature class table that was created in FDO. In fact, there were plenty of memory on my machine (more than 1-1.5G available out of 3.2 G in 32bit machine). According to one thread in this forum, the reason is memory is fragmented. I don't understand why FDO could not allocate dynamically linked memory.

Anyway, I am thinking a workaround solution that is to avoid IInsert but use oracle ODP.net to insert data to the table.

The feature class was created with some default property like "CLASSID", "REVISIONNUMBER" and unique "FEATID". My questions:
(1) Can I use ODP.net to insert values for those properties as well?
(2) Any FDO data should I update (for example sequence id related with "FEATID" )?
(3) Any other complications?


Reply | Threaded
Open this post in threaded view
|

Re: Memory issue with FDO IInsert and workaround

Jackie Ng
1. Are you closing the feature readers returned by each insert execution?
2. Are you clearing and re-populating the property value collection of the insert command with new PropertyValue objects -or- are you nulling all the PropertyValue objects in the collection and re-setting the appropriate values for the next record to be inserted?

Point #2 is pertinent because the former approach results in n x m memory allocations (n property values, m records to be inserted), whereas the latter only requires n memory allocations. This may be the cause of the memory fragmentation.

- Jackie
Ray
Reply | Threaded
Open this post in threaded view
|

Re: Memory issue with FDO IInsert and workaround

Ray
Thanks Jackie for your input.

For #1: I simply call the following to do the insertion:
IInsertCmd.Execute().
Instead of the above, should I do the following?
IInsertCmd.Execute()
fdoReader.close()
Btw, I reuse the same IInsertCmd for all the insertions.

For #2, I use the former approach because I have to insert for a number of featureclasses and each of them have a dozen of properties. Instead of caching all the propertyvalues for each the featureclasses which will be hundred of propertyvalues in total, I wrote some generic code (a dozen lines of code) which works for all feature classes. So I did new propertyvalue for each record, but I dispose them after each insertion.

So FDO can not use fragmented memory? I called ODP.net which doesn't seem to be a problem.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issue with FDO IInsert and workaround

Romica Dascalescu
Hi Ray,

You should use the insert command like below:

using (FdoIFeatureReader fdoReader=IInsertCmd.Execute())
{
   fdoReader.Close();
}

Moreover in your code you should avoid calling FDO objects like:
FdoObj.Acollection.Add(*);
You should use it liek below:
collection = FdoObj.Acollection
collection.Add(*);

You can also try to call Dispose (or add using blocks) on new FDO objects to avoid GC to free them when is too late.

Regards,
Romy
________________________________________
From: [hidden email] [[hidden email]] on behalf of Ray [[hidden email]]
Sent: Tuesday, April 10, 2012 9:54 AM
To: [hidden email]
Subject: [fdo-users] Re: Memory issue with FDO IInsert and workaround

Thanks Jackie for your input.

For #1: I simply call the following to do the insertion:
IInsertCmd.Execute().
Instead of the above, should I do the following?
IInsertCmd.Execute()
fdoReader.close()
Btw, I reuse the same IInsertCmd for all the insertions.

For #2, I use the former approach because I have to insert for a number of
featureclasses and each of them have a dozen of properties. Instead of
caching all the propertyvalues for each the featureclasses which will be
hundred of propertyvalues in total, I wrote some generic code (a dozen lines
of code) which works for all feature classes. So I did new propertyvalue for
each record, but I dispose them after each insertion.

So FDO can not use fragmented memory? I called ODP.net which doesn't seem to
be a problem.

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Memory-issue-with-FDO-IInsert-and-workaround-tp4717196p4752522.html
Sent from the FDO Users mailing list archive at Nabble.com.
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users