[gdal-dev] Fwd: Re: RFC 47 and Threading

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[gdal-dev] Fwd: Re: RFC 47 and Threading

Even Rouault-2
Was likely intended to be "Reply all"

----------  Message transmis  ----------

Sujet : Re: [gdal-dev] RFC 47 and Threading
Date : jeudi 28 août 2014, 19:20:01
De : Blake Thompson <[hidden email]>
À : Even Rouault <[hidden email]>

Even and Andre,

> I want to start off by saying a big thanks to Blake for taking his time
> > to tackle what can only be a very difficult problem.
> >  From what I can observe, the current discussion seems to be around the
> > boundary of who should be responsible for ensuring thread safety around
> > the block cache. The core of GDAL versus the individual drivers.
> The core will necessarily have to know about thread-safety because the
> block
> cache is there. The discussion is more whether the drivers must also
> necessary
> be thread aware, or if core mechanisms are sufficient to hide this detail
> to the
> drivers. And potentially offering to drivers a mechanism to deal
> themselves with
> thread-safety if they can have a more optimized implementation than the
> default
> one.

Agreed, most specifically how to hide the detail of the protection of the
pointer to the block cache's data that is passed through IReadBlock,
IWriteBlock, and IRasterIO.

> > While I
> > see why such a conversation is important, as far as I am concerned, the
> > most important part should be how it affects users of GDAL at the
> > interface level.
> Agreed.
> > That is, if an application that is threaded is trying
> > to use GDAL, how does it ensure thread-safety? What you have to keep in
> > mind is that having some parts of the library not thread-safe basically
> > just pushes the mutexing/locking to the calling applications.
> Not necessarily. That's what I suggested in my previous email : if the
> costs of
> the mutex are not too expensive for non-threaded usage, then the API could
> systematically return thread-safe versions, that are potentially wrapped by
> GDALDatasetThreadSafe

So in the scope of my RFC, I am not certain how we could have a thread safe
cache and a non-threadsafe cache in any simple manner. I know that you are
specifically talking about the possiblity of thread safe datasets, which I
feel is necessarily part of this discussion but wanted to separate the two.
If the thread-safe cache is too expensive I feel like that is a major issue
however, and I am doing my best to avoid any performance hits for this

> >
> > Also, while it is important to document thread-safety limitations, might
> > I suggest adding thread-safe related capabilities (TestCapability),
> > especially if all drivers do not end-up having the same thread-safety
> > constraints.
> That might be a solution, although not ideal from a usability point of
> view (if
> we come with something more complex that non-thread-safe vs thread-safe,
> that
> might be difficult to understand by users of GDAL API), and from a
> GDAL-developer point of view as well (need to assess thread-safety for each
> driver).
> There can have subtelties : imagine that the VRT driver is made to be
> thread
> safe, but uses sources of drivers that might be not thread safe...
> >
> > I personally do not see a GDALDatasetThreadSafe wrapper as adding much
> > complexity. For instance, if you were to add a capability that indicates
> > if a driver is inherently thread-safe, you could add a new open method
> > to open a dataset in a thread-safe way with something like the following
> > pseudocode:
> >
> > DataSet OpenThreadSafe(GDALOpenInfo openInfo)
> > {
> >      DataSet dataSet = Open(openInfo);
> >
> >      if (!dataSet.TestCapability(THREAD_SAFE))
> >      {
> >           dataSet = wrapWithThreadSafeWrapper(dataSet);
> >      }
> >
> >      return dataSet;
> > }
> Yes, that's similar to what I imagined with my idea of GDAL_OF_THREADSAFE
> open
> flag.

On the topic of the thread safe wrapper, I have spent more time thinking
about it and I think this is probably the best solution to the problem of
making all Datasets read safe, and I am willing to champion another RFC to
implement this. However, the scope of this is even larger because it should
be required to work with the OGR datasets as well.



Spatialys - Geospatial professional services
gdal-dev mailing list
[hidden email]