Open Microscopy Environment

The OME community
https://www.openmicroscopy.org/community/

generating/attaching/retrieving a non-OMERO related file

https://www.openmicroscopy.org/community/viewtopic.php?f=6&t=595

Page 1 of 2

generating/attaching/retrieving a non-OMERO related file

PostPosted: Fri Jan 21, 2011 6:42 pm
by bhcho
Hi All,

I'm trying to generate an output file from my Script in OMERO.Insight and attach it to a dataset.
Later from another Script, I also want to retrieve that output file, given the dataset ID.

Could anyone help me about this?
I'm trying to generate a pickle file that packs a complex data as the following python snippet

Code: Select all
# for generating a pickle file
import pickle
output = open('model.pkl','wb')
pickle.dump(final_model, output)  # final_model: a complex data variable
output.close()

# for retrieving the pickle file
import pickle
pkl_file = open('model.pkl','rb')
final_model2 = pickle.load(pkl_file)


BK

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Mon Jan 24, 2011 3:59 pm
by wmoore
Hi,

This is quite common behavior.
All the 'Figure' scripts E.g. 'Thumbnail Figure' save their output as attachments to E.g Image or Dataset.
In fact, you can do it with a single call: scriptUtil.uploadAndAttachFile(), documented here http://hudson.openmicroscopy.org.uk/job ... AttachFile

Below is some sample code from Thumbnail Figure script. See http://trac.openmicroscopy.org.uk/omero ... _Figure.py

Code: Select all
import omero.util.script_utils as scriptUtil
format = "image/png"

parent = gateway.getDataset(Id, False)    # parent can be omero.model.ImageI or DatasetI or ProjectI

output = "thumbnailFigure.png"   
figure.save(output, "PNG")        # simply writes to a file in the current directory

# uploads the file to the server, attaching it to the 'parent' Project/Dataset as an OriginalFile annotation,
# with the figLegend as the description. Returns the fileAnnotation object
fileAnnotation = scriptUtil.uploadAndAttachFile(queryService, updateService, rawFileStore, parent, output, format, figLegend)

client.setOutput("File_Annotation", robject(fileAnnotation))


If you want to be able to identify the file uniquely E.g. from your second script, you could give the file a meaningful namespace. E.g. 'omero.constants.metadata.NSMOVIE' (see the method docs above).

To download the file again, you can use the downloadFile(), also from the script_utils package.
http://hudson.openmicroscopy.org.uk/job ... wnloadFile
But you first need to retrieve your file - probably using the queryservice.

I just cobbled together this example, so I'm not sure it works - but hopefully you should be able to work the rest out for yourself.

Code: Select all
dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = %s and " \
                    "fileAnn.ns = %s" % (datasetId, yourNameSpace) , None)
                   
originalFile = dal.child.file
           
for file in dataset.
filePath = scriptUtil.downloadFile(rawFileStore, originalFile)
print "Saved file at:", filePath
f = open(filePath)


Hope that helps,

Will

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Mon Jan 24, 2011 5:44 pm
by bhcho
Hi Will,

Thanks so much for the answer.
The attachment was successful, but for the retrieving, i have the following error.
I used the namespace of "omero.constants.metadata.CLASSIFIER" for the file.
and the dataset ID 452 clearly has the attached file of "final_model.pkl" when I look at the OMERO.Insight

Could you figure out which part has the problem with my code?
the code is
Code: Select all
            dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = %s and " \
                    "fileAnn.ns = %s" % (ClassfierdatasetID, namesp_classfier) , None)


Code: Select all
        * Traceback (most recent call last):
        *   File "./script", line 399, in <module>
        *     runAsScript()
        *   File "./script", line 367, in runAsScript
        *     "fileAnn.ns = %s" % (ClassfierdatasetID, namesp_classfier) , None)
        *   File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero_api_IQuery_ice.py", line 132, in findByQuery
        *     return _M_omero.api.IQuery._op_findByQuery.invoke(self, ((query, params), _ctx))
        * omero.QueryException: exception ::omero::QueryException
        * {
        *     serverStackTrace = ome.services.query.QueryException: Illegal query:select dal from DatasetAnnotationLink as dal join fetch dal.child as fileAnn join dal.parent as dataset where dataset.id = 452 and fileAnn.ns = omero.constants.metadata.CLASSIFIER
        * Invalid path: 'omero.constants.metadata.CLASSIFIER' [select dal from ome.model.annotations.DatasetAnnotationLink as dal join fetch dal.child as fileAnn join dal.parent as dataset where dataset.id = 452 and fileAnn.ns = omero.constants.metadata.CLASSIFIER]
        *       at ome.services.query.StringQuery.buildQuery(StringQuery.java:60)
        *       at ome.services.query.Query.doInHibernate(Query.java:222)
        *       at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
        *       at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
        *       at ome.logic.QueryImpl.execute(QueryImpl.java:141)
        *       at ome.logic.QueryImpl.findByQuery(QueryImpl.java:337)
        *       at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)
        *       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        *       at java.lang.reflect.Method.invoke(Method.java:597)
        *       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        *       at ome.security.basic.EventHandler.invoke(EventHandler.java:157)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:231)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:111)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        *       at $Proxy52.findByQuery(Unknown Source)
        *       at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)
        *       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        *       at java.lang.reflect.Method.invoke(Method.java:597)
        *       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        *       at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:83)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:40)
        *       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        *       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        *       at $Proxy52.findByQuery(Unknown Source)
        *       at sun.reflect.GeneratedMethodAccessor579.invoke(Unknown Source)
        *       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        *       at java.lang.reflect.Method.invoke(Method.java:597)
        *       at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179)
        *       at ome.services.throttling.Callback.run(Callback.java:56)
        *       at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
        *       at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:136)
        *       at ome.services.blitz.impl.QueryI.findByQuery_async(QueryI.java:92)
        *       at omero.api._IQueryTie.findByQuery_async(_IQueryTie.java:113)
        *       at omero.api._IQueryDisp.___findByQuery(_IQueryDisp.java:342)
        *       at omero.api._IQueryDisp.__dispatch(_IQueryDisp.java:508)
        *       at IceInternal.Incoming.invoke(Incoming.java:159)
        *       at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
        *       at Ice.ConnectionI.message(ConnectionI.java:972)
        *       at IceInternal.ThreadPool.run(ThreadPool.java:577)
        *       at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
        *       at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
        *
        *     serverExceptionClass = ome.services.query.QueryException
        *     message = Illegal query:select dal from DatasetAnnotationLink as dal join fetch dal.child as fileAnn join dal.parent as dataset where dataset.id = 452 and fileAnn.ns = omero.constants.metadata.CLASSIFIER
        * Invalid path: 'omero.constants.metadata.CLASSIFIER' [select dal from ome.model.annotations.DatasetAnnotationLink as dal join fetch dal.child as fileAnn join dal.parent as dataset where dataset.id = 452 and fileAnn.ns = omero.constants.metadata.CLASSIFIER]
        * }
        *

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Mon Jan 24, 2011 6:57 pm
by jmoore
BK,

in the formatting two apostrophes apparently got lost. Try:
Code: Select all
            dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = %s and " \
                    "fileAnn.ns = '%s' " % (ClassfierdatasetID, namesp_classfier) , None)


In general, though, it is safer to use Parameters:

Code: Select all
            params = omero.sys.ParametersI()
            params.addLong("id", ClassifierdatasetID);
            params.addString("ns", namesp_classfier)
            dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = :id and " \
                    "fileAnn.ns = :ns ", params)


Cheers,
~Josh

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Mon Jan 24, 2011 9:40 pm
by bhcho
Hi Josh,

Thanks, and now I got the following error.
my code is as follows.
I'm sorry about so many detailed questions.

BK

Code: Select all
            params = omero.sys.ParametersI()
            params.addLong("id", ClassfierdatasetID);
            params.addString("ns", namesp_classfier)

            dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                    "join fetch dal.child as fileAnn " \
                    "join dal.parent as dataset " \
                    "where dataset.id = :id and " \
                    "fileAnn.ns = :ns ", params)

            originalFile = dal.child.file
           
            filePath = scriptUtil.downloadFile(rawFileStore, originalFile)
            print "Saved file at:", filePath
            f = open(filePath)




* filePath = scriptUtil.downloadFile(rawFileStore, originalFile)
* File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero/util/script_utils.py", line 245, in downloadFile
* fileSize = originalFile.getSize().getValue()
* File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero_model_OriginalFileI.py", line 294, in getSize
* self.errorIfUnloaded()
* File "/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero_model_OriginalFileI.py", line 47, in errorIfUnloaded
* raise _omero.UnloadedEntityException("Object unloaded:"+str(self))
* omero.UnloadedEntityException: Object unloaded:object #0 (::omero::model::OriginalFile)
* {
* _id = object #1 (::omero::RLong)
* {
* _val = 7026
* }
* _details = <nil>
* _loaded = False
* _version = <nil>
* _pixelsFileMapsSeq =
* {
* }
* _pixelsFileMapsLoaded = False
* _pixelsFileMapsCountPerOwner =
* {
* }
* _path = <nil>
* _size = <nil>
* _atime = <nil>
* _mtime = <nil>
* _ctime = <nil>
* _sha1 = <nil>
* _mimetype = <nil>
* _annotationLinksSeq =
* {
* }
* _annotationLinksLoaded = False
* _annotationLinksCountPerOwner =
* {
* }
* _name = <nil>
* }
*

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Mon Jan 24, 2011 9:56 pm
by wmoore
Hi,

I think you need to load (fetch) the file in the query.
Try adding "join fetch fileAnn.file" in this context...

Code: Select all
join fetch dal.child as fileAnn
join fetch fileAnn.file
join dal.parent as dataset


See if that works!

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Mon Jan 24, 2011 10:18 pm
by bhcho
Thanks Will and Josh,

It finally works!!!!

BK

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Tue Jan 25, 2011 4:17 pm
by wmoore
Final thought: You should probably choose a nicer namespace that would enable someone else getting info from your server to know what application etc the data came from. This avoids confusion with namespaces that are part of the OMERO clients' behavior etc.

Something like:

AppName.scriptName.Variable

Whatever seems sensible to you!

Will.

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Fri May 06, 2011 6:45 pm
by bhcho
Hi all,

I'd like to ask a follow-up question.
Currently, I'm retrieving the non-OMERO attachment file from a dataset by
(for a specific namespace)
Code: Select all
                params = omero.sys.ParametersI()
                params.addLong("id", My_datasetID);
                params.addString("ns", My_namesp)
                dal = queryService.findByQuery("select dal from DatasetAnnotationLink as dal " \
                        "join fetch dal.child as fileAnn " \
                        "join fetch fileAnn.file " \
                        "join dal.parent as dataset " \
                        "where dataset.id = :id and " \
                        "fileAnn.ns = :ns ", params)
                originalFile = dal.child.file
                filePath = scriptUtil.downloadFile(rawFileStore, originalFile)


1.
it seems like "originalFile" is a file object. but how can I get the file name?

2.
If there are multiple files with the same namespace, how can I get all the files at once (and their filenames)?

3.
how can I retrieve just one file with a specific file name?


thanks in advance,
BK

Re: generating/attaching/retrieving a non-OMERO related file

PostPosted: Fri May 06, 2011 7:41 pm
by jmoore
Hi BK,

bhcho wrote:1. it seems like "originalFile" is a file object. but how can I get the file name?


omero.model.OriginalFile objects have a "name" and a "path" field. You may also want to use "mimetype".

2. If there are multiple files with the same namespace, how can I get all the files at once (and their filenames)?


Use something like:
Code: Select all
                        "join fetch fileAnn.file as file " \
                        "join dal.parent as dataset " \
                        "where dataset.id = :id and " \
                        "fileAnn.ns = :ns and file.name = :name", params)


3. how can I retrieve just one file with a specific file name?


What do you mean? findByQuery always returns on element.

thanks in advance,
BK


Cheers,
~Josh
All times are UTC
Page 1 of 2
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/