Archives

Anticipation

    No dates present

Unmanaged callbacks across AppDomains

In one of the projects I’m working on I hit a fairly nasty problem involving (as might be obvious from the title) both AppDomains and unmanaged code calling back into managed code. But first, a little background.

It all started with an unmanaged C++ class library. We’ve been using it for a while from other unmanaged C++ applications, but since we’ve been writing most new applications in C#.NET, the time had come to bring the library (kicking and screaming) into the .NET world. Naturally, the way forward was to write a compatibility layer in C++.NET, thereby taking advantage of its IJW framework (which makes C++ the best .NET language to use to interface with native code). (.NET 2.0, of course. Don’t even get me started about some of the problems interfacing native & managed code in 1.x.)

One of the key ingredients was an unmanaged callback. Since some of the work of the unmanaged base library was asynchronous, we gave it an object to call back through in order to tell us when the results were ready — meaning that the unmanaged code needs to call into the managed code. This is normally simple enough — while unmanaged classes can’t hold on to managed objects directly, they can through a GCHandle (or the equivalent helper class, gcroot). The code went something like this (paraphrased to protect the guilty):

class MyCallback : public INativeCallback
{
public:
  explicit MyCallback(ManagedClass^ managed) : m_Managed(managed)
  {
  }
 
  virtual void OnNativeCallback(const BYTE *data, DWORD length)
  {
    cli::array<Byte>^ bytes = gcnew cli::array<Byte>(length);
    Marshal::Copy(IntPtr(const_cast<Byte*>(data)), bytes, 0, length);
    ManagedData^ mdata = gcnew ManagedData(bytes);
    m_Managed->OnCallback(mdata);
  }
 
private:
  gcroot<ManagedClass^> m_Managed;
};
 
ref class ManagedClass
{
public:
  ManagedClass()
  {
    m_Callback = new MyCallback(this);
    NativeFramework::RegisterCallback(m_Callback);
  }
 
  void OnCallback(ManagedData^ data)
  {
    // ...
  }
 
private:
  MyCallback *m_Callback;
};

So, all well and good so far, right? No such luck. At first this was all going smoothly; the code was coming together and manual testing of the application showed that it was communicating properly with the library and with native applications that were also using the library. Once the proof of concept was in place it was time to ensure nothing went wrong later on through the use of unit testing. This is where the problems began, however.

The unit tests in question were written using the NUnit framework, which runs the tests in a separate AppDomain. This is useful because it permits the test runner to treat the application/library under test as a plugin — keeping it loaded only while actually running the tests, thereby allowing it to be recompiled and have the tests run again without having to exit the test runner.

The problem is that AppDomains are purely a managed construct. They’re intended to keep sections of managed code mostly isolated from each other (as in the above plugin-like case), and as such objects usually only exist in one AppDomain at a time. Unmanaged code of course knows nothing about any of this. Consequently, when calling managed code from unmanaged code, the compiler has to pick one AppDomain to use, and it appears to pick the first one. This is fine for most applications, since normally apps only use one AppDomain — which is why this code was working at first.

But when running the unit tests, it was executing in a second AppDomain, and the callback failed. Specifically, when trying to access the m_Managed object (from the gcroot, which you’ll recall is a GCHandle) an exception was thrown saying “Cannot pass a GCHandle across AppDomains”.

The solution is to use delegates. They’re not just function pointers — they also contain an object reference, a few other odds and ends, and (most importantly for us) a reference to the AppDomain that created it. However there’s a catch. Delegates themselves are managed objects, and so would have to be stored in a gcroot if held in unmanaged code — and we already know that we can’t access anything in a gcroot outside its original AppDomain.

Fortunately there’s a loophole: delegates can be marshalled into unmanaged function pointers (via a thunking layer). There is a downside to this though. Since the method signatures must match as closely as possible, and since a purely unmanaged function pointer can’t have anything to do with managed objects, the parameters must be restricted to native types. This required a bit of a redesign, but fortunately since you’re coming in from unmanaged code all the data you’re dealing with is going to be native anyway. So here’s the redesigned code. It’s possible there’s still something that could be improved in it; but this one does the trick, and maybe it’ll help someone else who has been struggling with this issue :smile:

delegate void CallbackDelegate(IntPtr data, unsigned int length);
typedef void (__stdcall *CallbackNative)(const BYTE *data, DWORD length);
 
class MyCallback : public INativeCallback
{
public:
  explicit MyCallback(CallbackDelegate^ callDelegate)
      : m_Delegate(callDelegate)
  {
    m_Native = (CallbackNative)
        Marshal::GetFunctionPointerForDelegate(callDelegate).ToPointer();
  }
 
  virtual void OnNativeCallback(const BYTE *data, DWORD length)
  {
    m_Native(data, length);
  }
 
private:
  gcroot<CallbackDelegate^> m_Delegate;
  CallbackNative m_Native;
};
 
ref class ManagedClass
{
public:
  ManagedClass()
  {
    CallbackDelegate^ callback = gcnew CallbackDelegate(this,
        &ManagedClass::OnNativeCallback);
    m_Callback = new MyCallback(callback);
    NativeFramework::RegisterCallback(m_Callback);
  }
 
  ~ManagedClass()
  {
    this->!ManagedClass();
  }
 
  !ManagedClass()
  {
    NativeFramework::UnregisterCallback(m_Callback);
    delete m_Callback;
    m_Callback = NULL;
  }
 
  void OnCallback(ManagedData^ data)
  {
    // ...
  }
 
private:
  void OnNativeCallback(IntPtr data, unsigned int length)
  {
    cli::array<Byte>^ bytes = gcnew cli::array<Byte>(length);
    Marshal::Copy(data, bytes, 0, length);
    ManagedData^ mdata = gcnew ManagedData(bytes);
    OnCallback(mdata);
  }
 
private:
  MyCallback *m_Callback;
};

(Don’t forget to Dispose the ManagedClass when you’re done with it!)

And there we have it; my first Programming post. I hope at least someone finds it interesting! :grin:

Note: I’ve since written a followup post, which you might find useful to read now that the above is fresh in your memory.

26 comments to Unmanaged callbacks across AppDomains

  • Ok, I’d like to get you started on Managed C++ in 1.x. :)

    Seriously, I’m having the exact same problem as you did and luckily I found your post. Unfortunately, our code is still in the old MCPP syntax, although I’ve recompiled it on VS 2005 with the oldSyntax switch. Do you think I could do something similar to the above with the old syntax?

    I’m not a C++ pro, so I’d rather not rewrite it in C++/CLI.

    Thanks,
    Jason

  • I haven’t tried it, but as far as I know anything you can do with the new syntax you can also do in the old syntax — it’s just a lot more verbose (and has lots of double-underscores). As long as you’re using VS2005, anyway (and targetting 2.0). (The above method won’t work in 1.x, since it uses functions added in 2.0 — but hybrid 1.x DLLs don’t work all that well anyway.)

    If you take my code above and backport the syntax (eg. replace ^s with *s, refs with __gcs, etc) then you should be close.

  • Franz

    Hi

    I have exactly the same Problem and your code looks very fine. But I have a problem with INativeCallback and NativeFramework::RegisterCallback.

    What is INativeCallback and NativeFramework?
    Where can I get it?

    One additional problem may be, my callback gets a very deep C++ class instead of an simple bytearray. But I think, that’s not so big problem :)

    Many thanks for your help!
    Franz

  • INativeCallback and NativeFramework are placeholders for the native classes you’re trying to interface to. In other words, that’s your external code, not part of .NET — so you’ve already got it :mrgreen:

    And no, there won’t be any issues with passing C++ classes through the callback, it’ll just work (bearing in mind what I said later). Though if you’re going to be interfacing with C# or VB code at some point then you should convert the data to managed classes along the way (after calling through the delegate; you can’t do it earlier).

  • Nelson Fung

    Excellent Article!!

    You saved my day (again even when we are no longer working together), Gavin!
    I was trying to use AppDomain to load a native dll which receives phone calls, as I want to have control of unloading and reloading the dll. I was stuck at the callbacks part where your article just helped me fixed it.

    Thank you so much. You are a real Guru!

  • [...] appdomain to select to execute? In general, no, it doesn’t — you must use delegates for that. See http://lambert.geek.nz/2007/05/29/un…main-callback/ — [...]

  • codethecoder

    Please post a link of full source code.

  • Why? All the relevant bits are there. The implementation details of the native framework aren’t interesting to the discussion.

    I am planning to make another post soon along similar lines to this one, showing a different technique which puts more emphasis on the native code. Perhaps that’ll be more to your liking.

  • mike

    If ManagedClass was implemented in C#. how do I store the delegate in gcroot? is it automatic once i call somedelegate = new delegate(…)??
    do i use GCHandle.Alloc(somedelegate) ?

  • The gcroot isn’t the problem then — the problem is the native framework call and storing a reference to the unmanaged handler class (MyCallback).

    It’s not totally impossible for ManagedClass to be implemented in C#, but it’s far more hassle than it’s worth, at least for the scenario I’ve specified (implementing a C++ abstract callback class), since MyCallback has to be implemented in C++ anyway.

  • Thanks for that. It was just what I needed and the example code was a great help at moving me in the right direction.

  • dave dowd

    I had the same issue. Your example helped me – thanks. My example may help others – see below. I show my code (before) and (after).

    // LogAdapter.h
     
    #pragma once
     
    #include "vcclr.h"
    #include "ILog.h"
     
    namespace Rfa { namespace UnManaged {
    	class RdmLogMessage;
    }}
     
    namespace MarketData {
     
        /// Class to adapt unmanaged to managed logging.
        class LogAdapter
        {
     
        public:
     
            // Construct unmanaged class using managed object.
    		LogAdapter(ILog^ object)
    			: _object(object)
    		{
    		}
     
    		// Unmanaged to managed callback
    		void operator()(const Rfa::UnManaged::RdmLogMessage&amp;)
    		{
    			if (_object)
    			{
    				System::String^ text = Marshal::ToManagedString( msg.ToString() );
    				_object->Log( text );
    			}
    		}
     
        private:
     
    		// Restrictions
    		LogAdapter(const LogAdapter&);
    		LogAdapter&amp; operator=(const LogAdapter&);
     
    		gcroot _object;
     
        };
     
    }

    My code after:

    // LogAdapter.h
     
    #pragma once
     
    #include "vcclr.h"
    #include "LogDelegateAdapter.h"
     
    namespace Rfa { namespace UnManaged {
    	class RdmLogMessage;
    }}
     
    namespace MarketData {
     
    	/// Native logging interface adapter class.
    	/// Adapts RdmLogger to LogAdapter.
    	class LogAdapter
    	{
     
    	public:
     
    		/// Construct using (managed) ILog instance.
    		/// Scope of LogDelegateAdapter is managed here.
    		LogAdapter(ILog^ logger)
    			: _logDelegateAdapter(gcnew LogDelegateAdapter(logger))
    			, _function_pointer(_logDelegateAdapter->FunctionPointer)
    		{
    		}
     
    		/// Construct using LogDelegateAdapter's (native) function pointer.
    		/// 
    		/// RdmLogger messages are forwarded to this function.
    		/// Scope of LogDelegateAdapter is managed elsewhere.
    		/// 
    		LogAdapter(LogDelegateAdapter::function_pointer function_pointer)
    			: _logDelegateAdapter(nullptr)
    			, _function_pointer(function_pointer)
    		{
    		}
     
    		/// RdmLogger callback functor.
    		/// 
    		/// Both RdmConsumer and RdmAuthenticator use RdmLogger.
    		/// RdmLogger sends log messages via this functor.
    		/// Upon receipt, they are forwarded to the log delegate function.
    		/// 
    		void operator()(const Rfa::UnManaged::RdmLogMessage& message)
    		{
    			// If we have any problem calling into the AppDomain, disable the callback.
    			// This might occur with multiple AppDomains and the one we're using is being unloaded.
    			try
    			{
    				if (_function_pointer)
    					(*_function_pointer)(message.ToString().c_str());
    			}
    			catch (...)
    			{
    				_function_pointer = 0;
    			}
    		}
     
    	private:
     
    		// Restrictions
    		LogAdapter(const LogAdapter&);
    		LogAdapter& operator=(const LogAdapter&);
     
    		// LogDelegateAdapter instance
    		gcroot          _logDelegateAdapter;
    		// Native entry point to (managed) LogDelegateAdapter::Log(...).
    		LogDelegateAdapter::function_pointer _function_pointer;
     
    	};
     
    }
     
    // LogDelegateAdapter.h
     
    #pragma once
     
    #include "ILog.h"
     
    namespace MarketData {
     
    	using namespace System;
     
    	/// Adapter Class enabling native to managed callbacks.
    	/// 
    	/// This class uses the delegate pattern to support native to managed callbacks.
    	/// The delegate pattern supports managed callbacks whether or not the callback
    	/// is to the default AppDomain.  Normally, there is just one AppDomain, however,
    	/// when using NUnit, tests do not run in the default AppDomain.
    	/// 
    	private ref class LogDelegateAdapter
    	{
     
    	public:
     
    		/// The log delegate definition.
    		delegate void LogDelegate(IntPtr ptr);
     
    		/// The log delegate function_pointer definition.
    		typedef void (__stdcall *function_pointer)(const char*);
     
    		/// Construct log delegate adapter using ILog instance.
    		LogDelegateAdapter(ILog^ logger)
    			: _logger(logger)
    		{
    			_logDelegate = gcnew LogDelegate(this, &LogDelegateAdapter::Log);
    			_function_pointer = static_cast(System::Runtime::InteropServices::Marshal::GetFunctionPointerForDelegate(_logDelegate).ToPointer());
    		}
     
    		/// The function_pointer points to me.
    		void Log(IntPtr ptr)
    		{
    			const char* c_str = static_cast(ptr.ToPointer());
    			String^ textMessage = gcnew String(c_str);
    			_logger->Log(textMessage);
    		}
     
    		/// Get the function_pointer.
    		property function_pointer FunctionPointer
    		{
    			function_pointer get()
    			{
    				return _function_pointer;
    			}
    		}
     
    		/// The ILog instance.
    		property ILog^ Logger
    		{
    			ILog^ get()
    			{
    				return _logger;
    			}
    		}
     
    	private:
     
    		// Log messages are (ultimately) sent to ILog
    		ILog^            _logger;
    		// The log delegate is: this->Log(IntPtr)
    		LogDelegate^     _logDelegate;
    		// The log delegate native entry point.
    		function_pointer _function_pointer;
     
    	};
     
    }
  • Ok, but you can just pass the std::string (or your Rfa::UnManaged::RdmLogMessage) through the function pointer and delegate (so it winds up in LogDelegateAdapter::Log). You don’t need to get a char* and use IntPtr.

    (The downside of doing that is that you then won’t be able to call LogDelegateAdapter::Log from fully managed code, but then usually you don’t need to anyway. And if you do, then you can just define another overload.)

  • Error: Cannot pass a GCHandle across AppDomains…

    I’m currently working on adding easy to use Real Time Data server support to my Managed XLL Excel Addin system. This lets you use the =RTD() functionality of Excel to push real time data into your spreadsheets without needing to……

  • great ! it worked for me as well
    .Net framework 3.5 does not have NativeFramework::

    Thanks you :-)

  • [...] This link is about creating a callback from unmanaged code into an AppDomain using a thunking trick. I’m not sure this can help you but maybe you’ll find this useful to create some kind of a workaround. [...]

  • Pass C++/CLI delegate as function pointer into native C++ class…

    To Pass C++/CLI delegate as function pointer into native C++ class, you can make use of this .net method: System::Runtime::InteropServices::Marshal::GetFunctionPointerForDelegate, the key point is: define the delagate (MyDelegate^ m_myDelegate in my ex…

  • Osvaldo

    I want to thank you for publishing this article. I had exactly this problem and would not have solved without your explanations and your sample code. I have seen other articles, but this is the most illustrative. Many, thank you very much!

  • diane

    Thanks you so much. You save my day. That was exactly what I needed !!!

  • [...] diane on: Unmanaged callbacks across AppDomains [...]

  • APA

    Thanks sooo much.
    It worked for us.

  • Chris

    Thank you so much!!!!! Best article!

  • Stephen Tuggy

    @dave dowd:
    Quick note: In LogDelegateAdapter.h (after), you have the following line near the top:

    using namespace System;

    Doesn’t that pollute the global namespace when you include it in a header file?

    Thanks for listing your code in detail like this. It is helping me figure out how to tackle my particular situation.

  • Ganesh Sittampalam

    I’ve managed to work round the same problem by using the COM Callable Wrapper for the managed object works: instead of storing a gcroot<ManagedObject>, store the pointer as an IUnknown * using GetIUnknownForObject, and do the reverse translation with GetObjectForIUnknown before making the callback.

    I presume that the COM Callable Wrapper must wrap up the AppDomain information, because COM knows nothing about AppDomains itself. Certainly it seems to work in practice.

    The downside is losing a little bit of type safety because IUnknown * loses the actual object type and you later have to downcast, but that seems like a small price to pay for the greater simplicity.

    (I also posted about this on StackOverflow)

  • Stephen: no, it doesn’t pollute the global namespace, it pollutes the MarketData namespace. But yes, in my own code I usually try to avoid doing that sort of thing even though it makes headers more wordy.

  • Hi there, I want to subscribe for this blog to obtain most up-to-date
    updates, therefore where can i do it please assist.

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">