How to attach/detach a property from a propertyvalues of IInserCmd

classic Classic list List threaded Threaded
9 messages Options
Ray
Reply | Threaded
Open this post in threaded view
|

How to attach/detach a property from a propertyvalues of IInserCmd

Ray
I have an insert cmd of a feature class which has about 100 properties. Now I have around 1 million record to populate. If I NEW-ed property and property values for each record, FDO will bump into insufficient memory issue due to fragemented memory but not really insufficient memory.

So I like to use the same insertcmd and propertyvalues and use the dictionary strategy that Jackie Ng recommened.

The question is that some of the records have some NULL property values. So I remove the property from oPropertyValues for the record, then add it back right away. However, after a few thousand records, it starts to give me read/write protected memory error.

The following is simiplified code snippet. Can anyone give me any suggestions? I wonder if the propertyvalues.attach and detach() method can be used for this purpose and how I use them?


oInsertCmd = oFDOConnection.CreateCommand(CommandType.CommandType_Insert)
            oInsertCmd.SetFeatureClassName(fcName)
            oPropertyValues = oInsertCmd.PropertyValues

.............
'Remove property value from oPropertyValues if it has NULL value

If fdoReader.IsNull(propertyName) Then
                        oPropertyValue = oPropertyValues.GetItem(propertyName)
                        nullProps.Add(propertyName, oPropertyValue)
                        oPropertyValues.Remove(oPropertyValue)
End If

.......................

'Add property value back to oPropertyValues for next record
Dim npenum As IEnumerator
                If nullProps.Count > 0 Then
                    npenum = nullProps.Values.GetEnumerator()
                    While npenum.MoveNext
                  If oPropertyValues.Contains(npenum.Current) = False Then
                            oPropertyValues.Add(npenum.Current)
                        End If
                    End While
                End If
   
 nullProps.Clear()

Thanks,
Ray

Reply | Threaded
Open this post in threaded view
|

Re: How to attach/detach a property from a propertyvalues of IInserCmd

Greg Boone
Why remove the NULL property? Can't you just mark the insert property as containing a NULL value?

Greg

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Ray
Sent: Monday, July 09, 2012 10:24 AM
To: [hidden email]
Subject: [fdo-users] How to attach/detach a property from a propertyvalues of IInserCmd

I have an insert cmd of a feature class which has about 100 properties. Now I have around 1 million record to populate. If I NEW-ed property and property values for each record, FDO will bump into insufficient memory issue due to fragemented memory but not really insufficient memory.

So I like to use the same insertcmd and propertyvalues and use the dictionary strategy that Jackie Ng recommened.

The question is that some of the records have some NULL property values. So I remove the property from oPropertyValues for the record, then add it back right away. However, after a few thousand records, it starts to give me read/write protected memory error.

The following is simiplified code snippet. Can anyone give me any suggestions? I wonder if the propertyvalues.attach and detach() method can be used for this purpose and how I use them?


oInsertCmd = oFDOConnection.CreateCommand(CommandType.CommandType_Insert)
            oInsertCmd.SetFeatureClassName(fcName)
            oPropertyValues = oInsertCmd.PropertyValues

.............
'Remove property value from oPropertyValues if it has NULL value

If fdoReader.IsNull(propertyName) Then
                        oPropertyValue =
oPropertyValues.GetItem(propertyName)
                        nullProps.Add(propertyName, oPropertyValue)
                        oPropertyValues.Remove(oPropertyValue)
End If

.......................

'Add property value back to oPropertyValues for next record Dim npenum As IEnumerator
                If nullProps.Count > 0 Then
                    npenum = nullProps.Values.GetEnumerator()
                    While npenum.MoveNext
                  If oPropertyValues.Contains(npenum.Current) = False Then
                            oPropertyValues.Add(npenum.Current)
                        End If
                    End While
                End If
   
 nullProps.Clear()

Thanks,
Ray



--
View this message in context: http://osgeo-org.1560.n6.nabble.com/How-to-attach-detach-a-property-from-a-propertyvalues-of-IInserCmd-tp4986964.html
Sent from the FDO Users mailing list archive at Nabble.com.
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
Reply | Threaded
Open this post in threaded view
|

Re: How to attach/detach a property from a propertyvalues of IInserCmd

Romica Dascalescu
Yes, there is a method SetNull(true) for each value and avoid adding/removing property values.

Romy
________________________________________
From: [hidden email] [[hidden email]] on behalf of Greg Boone [[hidden email]]
Sent: Monday, July 09, 2012 10:38 AM
To: FDO Users Mail List
Subject: Re: [fdo-users] How to attach/detach a property from a propertyvalues  of IInserCmd

Why remove the NULL property? Can't you just mark the insert property as containing a NULL value?

Greg

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Ray
Sent: Monday, July 09, 2012 10:24 AM
To: [hidden email]
Subject: [fdo-users] How to attach/detach a property from a propertyvalues of IInserCmd

I have an insert cmd of a feature class which has about 100 properties. Now I have around 1 million record to populate. If I NEW-ed property and property values for each record, FDO will bump into insufficient memory issue due to fragemented memory but not really insufficient memory.

So I like to use the same insertcmd and propertyvalues and use the dictionary strategy that Jackie Ng recommened.

The question is that some of the records have some NULL property values. So I remove the property from oPropertyValues for the record, then add it back right away. However, after a few thousand records, it starts to give me read/write protected memory error.

The following is simiplified code snippet. Can anyone give me any suggestions? I wonder if the propertyvalues.attach and detach() method can be used for this purpose and how I use them?


oInsertCmd = oFDOConnection.CreateCommand(CommandType.CommandType_Insert)
            oInsertCmd.SetFeatureClassName(fcName)
            oPropertyValues = oInsertCmd.PropertyValues

.............
'Remove property value from oPropertyValues if it has NULL value

If fdoReader.IsNull(propertyName) Then
                        oPropertyValue =
oPropertyValues.GetItem(propertyName)
                        nullProps.Add(propertyName, oPropertyValue)
                        oPropertyValues.Remove(oPropertyValue)
End If

.......................

'Add property value back to oPropertyValues for next record Dim npenum As IEnumerator
                If nullProps.Count > 0 Then
                    npenum = nullProps.Values.GetEnumerator()
                    While npenum.MoveNext
                  If oPropertyValues.Contains(npenum.Current) = False Then
                            oPropertyValues.Add(npenum.Current)
                        End If
                    End While
                End If

 nullProps.Clear()

Thanks,
Ray



--
View this message in context: http://osgeo-org.1560.n6.nabble.com/How-to-attach-detach-a-property-from-a-propertyvalues-of-IInserCmd-tp4986964.html
Sent from the FDO Users mailing list archive at Nabble.com.
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
Reply | Threaded
Open this post in threaded view
|

Re: How to attach/detach a property from a propertyvalues of IInserCmd

Romica Dascalescu
SetNull() without any parameter.

Romy
________________________________________
From: [hidden email] [[hidden email]] on behalf of Romica Dascalescu [[hidden email]]
Sent: Monday, July 09, 2012 10:43 AM
To: FDO Users Mail List
Subject: Re: [fdo-users] How to attach/detach a property from a propertyvalues  of IInserCmd

Yes, there is a method SetNull(true) for each value and avoid adding/removing property values.

Romy
________________________________________
From: [hidden email] [[hidden email]] on behalf of Greg Boone [[hidden email]]
Sent: Monday, July 09, 2012 10:38 AM
To: FDO Users Mail List
Subject: Re: [fdo-users] How to attach/detach a property from a propertyvalues  of IInserCmd

Why remove the NULL property? Can't you just mark the insert property as containing a NULL value?

Greg

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Ray
Sent: Monday, July 09, 2012 10:24 AM
To: [hidden email]
Subject: [fdo-users] How to attach/detach a property from a propertyvalues of IInserCmd

I have an insert cmd of a feature class which has about 100 properties. Now I have around 1 million record to populate. If I NEW-ed property and property values for each record, FDO will bump into insufficient memory issue due to fragemented memory but not really insufficient memory.

So I like to use the same insertcmd and propertyvalues and use the dictionary strategy that Jackie Ng recommened.

The question is that some of the records have some NULL property values. So I remove the property from oPropertyValues for the record, then add it back right away. However, after a few thousand records, it starts to give me read/write protected memory error.

The following is simiplified code snippet. Can anyone give me any suggestions? I wonder if the propertyvalues.attach and detach() method can be used for this purpose and how I use them?


oInsertCmd = oFDOConnection.CreateCommand(CommandType.CommandType_Insert)
            oInsertCmd.SetFeatureClassName(fcName)
            oPropertyValues = oInsertCmd.PropertyValues

.............
'Remove property value from oPropertyValues if it has NULL value

If fdoReader.IsNull(propertyName) Then
                        oPropertyValue =
oPropertyValues.GetItem(propertyName)
                        nullProps.Add(propertyName, oPropertyValue)
                        oPropertyValues.Remove(oPropertyValue)
End If

.......................

'Add property value back to oPropertyValues for next record Dim npenum As IEnumerator
                If nullProps.Count > 0 Then
                    npenum = nullProps.Values.GetEnumerator()
                    While npenum.MoveNext
                  If oPropertyValues.Contains(npenum.Current) = False Then
                            oPropertyValues.Add(npenum.Current)
                        End If
                    End While
                End If

 nullProps.Clear()

Thanks,
Ray



--
View this message in context: http://osgeo-org.1560.n6.nabble.com/How-to-attach-detach-a-property-from-a-propertyvalues-of-IInserCmd-tp4986964.html
Sent from the FDO Users mailing list archive at Nabble.com.
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
Ray
Reply | Threaded
Open this post in threaded view
|

Re: How to attach/detach a property from a propertyvalues of IInserCmd

Ray
Hi Romy,

I don't see a SetNull() method in either PropertyValue or PropertyValues. I am using FDO 3.4 and .NET API.

Thanks,
Ray
Reply | Threaded
Open this post in threaded view
|

Re: How to attach/detach a property from a propertyvalues of IInserCmd

Romica Dascalescu
So we have:
public class DataValue : LiteralValue
{
  public virtual void SetNull();
}
public class GeometryValue : LiteralValue
{
  public void SetNull();
}

public class PropertyValue : Disposable
{
    public ValueExpression Value { get; set; }
}
public class Expression : Disposable
{
   public ExpressionItemType ExpressionType { get; }
}

PropertyValue mypVal=...;
ValueExpression vexp = mypVal.Value;
switch (vexp.ExpressionType)
{
    case ExpressionItemType_DataValue:
    {
       DataValue dv = vexp as DataValue;
       dv.SetNull();
    }
    break;
    case ExpressionItemType_GeometryValue:

    {
       GeometryValue gv = vexp as GeometryValue;
       gv.SetNull();
    }
    break;
}

Or in case FDO 3.4 do not have ExpressionItemType :

PropertyValue mypVal=...;
ValueExpression vexp = mypVal.Value;

DataValue dv = vexp as DataValue;
if (dv != null){
   dv.SetNull();
}else{
    GeometryValue gv = vexp as GeometryValue;
    if (gv != null)
        gv.SetNull();
}

Romy.
________________________________________
From: [hidden email] [[hidden email]] on behalf of Ray [[hidden email]]
Sent: Monday, July 09, 2012 11:03 AM
To: [hidden email]
Subject: Re: [fdo-users] How to attach/detach a property from a propertyvalues  of IInserCmd

Hi Romy,

I don't see a SetNull() method in either PropertyValue or PropertyValues. I
am using FDO 3.4 and .NET API.

Thanks,
Ray

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/How-to-attach-detach-a-property-from-a-propertyvalues-of-IInserCmd-tp4986964p4986975.html
Sent from the FDO Users mailing list archive at Nabble.com.
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
Ray
Reply | Threaded
Open this post in threaded view
|

Re: How to attach/detach a property from a propertyvalues of IInserCmd

Ray
In reply to this post by Ray

I think I may be able to do the following:

oPropertyValue.Value = Nothing

But oPropertyValue.value was cached in a dictionary. So if I do that, it modifis the value in the cache. Then for next record, I have to reinitiatize the value and reinsert it to the dictionary.

Really I want is to cache initial values for all property values. For record with null property, I like to skip inserting this property. For record with not null property, I like to reuse the cached initial value without having to reinit and new-ed the property every time.

Thanks,
Ray
Reply | Threaded
Open this post in threaded view
|

Re: How to attach/detach a property from a propertyvalues of IInserCmd

Jackie Ng
In reply to this post by Ray
You should bulk insert like so:

 * Execute source query, get feature reader
 * Iterate it's Class Definition to construct the initial property value collection for the insert command
 * While source reader has results
    * Call SetNull(true) on all boxed property values in the collection
    * Iterate through each property
       * If property is not null for this current reader, call SetValue() on the corresponding boxed property value, with the matching GetXXX() call on the source reader
    * Insert with this property value collection
    * Read next result in source reader. Repeat.

This strategy only results in n memory allocations (n = number of properties in the reader). Every reader loop iteration is just flipping null states and setting values, no need to add/remove property values in each iteration.

- Jackie
Reply | Threaded
Open this post in threaded view
|

Re: How to attach/detach a property from a propertyvalues of IInserCmd

Romica Dascalescu
In reply to this post by Ray
My proposal will be faster and will create less FDO objects needed to be released by GC...
As you see SetNull() usage you should do the same to set a value.
This way you memory usage footprint will be really small.

Romy.
________________________________________
From: [hidden email] [[hidden email]] on behalf of Ray [[hidden email]]
Sent: Monday, July 09, 2012 11:18 AM
To: [hidden email]
Subject: Re: [fdo-users] How to attach/detach a property from a propertyvalues  of IInserCmd

I think I may be able to do the following:

oPropertyValue.Value = Nothing

But oPropertyValue.value was cached in a dictionary. So if I do that, it
modifis the value in the cache. Then for next record, I have to reinitiatize
the value and reinsert it to the dictionary.

Really I want is to cache initial values for all property values. For record
with null property, I like to skip inserting this property. For record with
not null property, I like to reuse the cached initial value without having
to reinit and new-ed the property every time.

Thanks,
Ray


--
View this message in context: http://osgeo-org.1560.n6.nabble.com/How-to-attach-detach-a-property-from-a-propertyvalues-of-IInserCmd-tp4986964p4986980.html
Sent from the FDO Users mailing list archive at Nabble.com.
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users