- Published:September 12th, 2008
- Comments:38 Comments
- Category:Firebug, Planet Mozilla
I have been waiting for a long time to write about this. Since Bug 430155 (new nsHttpChannel interface to allow examination of HTTP data before it is passed to the channel's creator) is now fixed, it's possible to intercept HTTP traffic from within a Firefox extension!
This fix should be part of Firefox 3.0.4.
This possibility is crucial for Firebug (and also Adblock Plus and Firekeeper can benefit from it) as one of its features is showing a response of any HTTP request made by a page. Until now, Firebug has been using Firefox cache and XHR monitoring to retrieve the response and display it to the user. While this worked quite fine for XHR (it's not that hard to monitor XHR responses) there was a big issue with monitoring all the other network requests. The most visible problem was probably submit of an HTTP form using POST method (known as double-load problem).
In some cases, the HTTP response isn't cached (and don't forget that Firefox cache can be disabled entirely) and Firebug had to fire a second request in order to get the response again and show it to the user. Of course, this was a big problem for every server that executed any (e.g. database) related actions (these have been executed twice). Uff...
Finally, I could implement new and reliable cache within Firebug 1.3 based on nsITraceableChannel. So, Firebug can use it anytime the response-source is needed without an additional request, e.g. if the user wants to see a network response in Firebug's Net panel...
Check it out, I am recommending latest Firebug 1.3 from SVN and till the Firefox 3.0.3 isn't available you can download latest trunk build from here. Any comments truly appreciated!
For those who are more interested, I have written an example that demonstrates how the nsITraceableChannel can be used.
How to intercept HTTP traffic
The main purpose of the nsITraceableChannel interface is to register a stream listener (nsIStreamListener) into an HTTP channel and monitor all data in it.
In order to register the listener within every incoming HTTP channel we also need to observe http-on-examine-response event. This event is fired whenever a response is received from the server, but before any data are available.
This is how the observer can be registered.
const Ci = Components.interfaces;
var observerService = Cc["@mozilla.org/observer-service;1"]
.getService(Ci.nsIObserverService);
observerService.addObserver(httpRequestObserver,
"http-on-examine-response", false);
Of course, we should make sure it's also properly unregistered.
"http-on-examine-response");
Implementation of the request-observer should be as follows:
{
observe: function(aSubject, aTopic, aData)
{
if (aTopic == "http-on-examine-response")
{
}
},
QueryInterface : function (aIID)
{
if (aIID.equals(Ci.nsIObserver) ||
aIID.equals(Ci.nsISupports))
{
return this;
}
throw Components.results.NS_NOINTERFACE;
}
});
See more info about http-on-modify-request and http-on-examine-response here.
The next step is to implement the stream listener. The listener is realized as a wrapper of the original listener. So, don't forget to call it in each method. If multiple extensions register a listener using the nsITraceableChannel, a chain of listeners is created.
this.originalListener = null;
}
TracingListener.prototype =
{
onDataAvailable: function(request, context, inputStream, offset, count) {
this.originalListener.onDataAvailable(request, context, inputStream, offset, count);
},
onStartRequest: function(request, context) {
this.originalListener.onStartRequest(request, context);
},
onStopRequest: function(request, context, statusCode) {
this.originalListener.onStopRequest(request, context, statusCode);
},
QueryInterface: function (aIID) {
if (aIID.equals(Ci.nsIStreamListener) ||
aIID.equals(Ci.nsISupports)) {
return this;
}
throw Components.results.NS_NOINTERFACE;
}
}
The registration is made within the httpRequestObserver.observe method. Just set the listener using setNewListener method and remember the original listener.
{
if (aTopic == "http-on-examine-response") {
var newListener = new TracingListener();
aSubject.QueryInterface(Ci.nsITraceableChannel);
newListener.originalListener = aSubject.setNewListener(newListener);
}
}
That's it!
Here is yet another implementation of the TracingListener that copies all the incoming data.
function CCIN(cName, ifaceName) {
return Cc[cName].createInstance(Ci[ifaceName]);
}
// Copy response listener implementation.
function TracingListener() {
this.originalListener = null;
this.receivedData = []; // array for incoming data.
}
TracingListener.prototype =
{
onDataAvailable: function(request, context, inputStream, offset, count)
{
var binaryInputStream = CCIN("@mozilla.org/binaryinputstream;1",
"nsIBinaryInputStream");
var storageStream = CCIN("@mozilla.org/storagestream;1", "nsIStorageStream");
var binaryOutputStream = CCIN("@mozilla.org/binaryoutputstream;1",
"nsIBinaryOutputStream");
binaryInputStream.setInputStream(inputStream);
storageStream.init(8192, count, null);
binaryOutputStream.setOutputStream(storageStream.getOutputStream(0));
// Copy received data as they come.
var data = binaryInputStream.readBytes(count);
this.receivedData.push(data);
binaryOutputStream.writeBytes(data, count);
this.originalListener.onDataAvailable(request, context,
storageStream.newInputStream(0), offset, count);
},
onStartRequest: function(request, context) {
this.originalListener.onStartRequest(request, context);
},
onStopRequest: function(request, context, statusCode)
{
// Get entire response
var responseSource = this.receivedData.join("");
this.originalListener.onStopRequest(request, context, statusCode);
},
QueryInterface: function (aIID) {
if (aIID.equals(Ci.nsIStreamListener) ||
aIID.equals(Ci.nsISupports)) {
return this;
}
throw Components.results.NS_NOINTERFACE;
}
}
Update:
The receivedData array should be initialized in the constructor of TracingListener object or within the onStartRequest method, to avoid sharing it among more instances. See more details about this in a neat article written by Jonathan Fingland.
this.receivedData = [];
}
38 Comments
Thanks for the examples. I already looked at an implementation with a storage stream in the bug - but that seems very inefficient (especially the fact that you have to create a new storage stream for each chunk of data). I wonder whether the same can be done is a more straightforward way. Unfortunately, you cannot implement nsIInputStream in JavaScript.
" newListener.listener = request.setNewListener(newListener);"
Wouldn't that be:
" newListener.originalListener = request.setNewListener(newListener);"
?
Yep, you right. Fixed, thanks!
Honza
So does this fix for the bug 430115 mean that I can not only gain access to the HTTP response before it is parsed (e.g. JavaScript is executed) but also modify the text of the response (e.g. JavaScript content) so that the first time the Gecko engine parses / executes the JavaScript, it will be my modified version of the JavaScript instead of the original?
I have been looking for a solution like this for some time as it would help me enable web sites that have been specifically targeted for IE to prevent them from running in Mozilla.
Your answer is kindly appreciated...
@James: yes, this is possible. Just past whatever you want to the original stream listener within the onDataAvailable method.
Honza
Thanks for your answer. One other quick question. Will this allow me to get at async/ajax requests/responses?
@James: yes, this interface allows to intercept all incoming responses including XHRs.
Honza
Will firebug ever be able to time DNS resolution separately from HTTP requests? I've been looking through the Firefox source for a good place to get at this info and it doesn't look like you can by using any existing services.
In the above example, there is a piece of code.
request.QueryInterface(Ci.nsITraceableChannel);I presume 'request' is not an implicit object available in the observe : function(aSubject, aTopic, aData).
Where was a reference to request object established in the observe() ?
Do I miss something here ? As you could observe I am not too clear about this statement.
request.QueryInterface(Ci.nsITraceableChannel);@Sunil: fixed, thanks! When http-on-examine-response event is fired, the request object is passed into the observe method as a subject.
Honza
Hi, I check this solution and I changhe this line in:
aSubject.QueryInterface(Ci.nsITraceableChannel);
in
aSubject.QueryInterface(Components.interfaces.nsITraceableChannel);
Firefox 3.0.1 says:
Errore: Could not convert JavaScript argument arg 0 [nsISupports.QueryInterface] = NS_ERROR_XPC_BAD_CONVERT_JS
Where is my error?
tnx, cnt00
The nsITraceableChannel interface is quite new. What version of Firefox do you use? Try Firefox 3.0.4pre, http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0/ and let me know.
Honza
Sorry i use FF 3.0.1
Now i update my FF to 3.0.3
The error now is:
Error: aSubject.setNewListener is not a function
The line is:
newListener.originalListener = aSubject.setNewListener(newListener);
I don't understand because i see this meth in
http://mxr.mozilla.org/firefox/source/netwerk/base/public/nsITraceableChannel.idl
I have to use 3.0.4?
Tnx a lot!
Yes. The interface should be available since 3.0.4 (it should have been part of 3.0.3, but the patch didn't make it).
Honza
It works with 3.0.4pre!
Tnx!
Can we use this to get data for "http-on-modify-request"?
@Yogi: if by "data" you mean an HTTP response then yes.
Hi,
i need to modify ALL the http body when is completed.
Example in you code: i need to modify "responseSource" and not "data" inside method onDataAvailable.
It's possible? Sorry and tnx
In general the pattern of callbacks is onStartRequest, N*onDataAvailable, onStopRequest. I've instrumented the code though and see cases where onDataAvailable is called missing a surrounding onStartRequest or onStopRequest. I've instrumented the code to see this.
It's not what I expected. Is this normal behavior? And if so, what explains it?
@uctc73: What version of Firefox do you use? Do you have a test case I could also try?
Hi, admin:
when I replace the original html data with my own, a problem is that the relative url will be rendered as the original uri + relative url, which is wrong, if there any interface which can fix this instead writing additional code to do this.
Thanks
Thank you. Great tutorial.
Could you possibly be so kind as to create sample add-on for FF that do that? Like replacing for example " test " with " TEST "? It is for those of us who are not sophisticated programmers. I can create couple regexes, but that is maximum for me. Plus here: http://forums.mozillazine.org/viewtopic.php?f=19&t=1033795 at least two more guys(leska and addoner) needed same thing...
I've tried to re-use the code you provided above in a firefox extension and while the request information seems to be correct (originalURI value is correct) the data I get is not related. The data I get seems to be from the first http request after the browser was started, not the current request. Thanks for providing this, I feel like I'm 90% of the way to making this work, but stuck on this one problem. Any Ideas?
To answer my own question and point out a problem in the sample code
TracingListener.prototype =
{
originalListener: null,
receivedData: [], // array for incoming data.
//...skipped
}
this causes all instances of TracingListener to share the same array. The fix is
function TracingListener() {
this.receivedData = [];
}
TracingListener.prototype =
{
originalListener: null,
receivedData: null,
//...skipped
}
@Honza
I'm not sure why the registration of the TracingListener is happening in the observe() method though? The observe() method is called every time a request/response is received, I'd imaging we'd just register the Listener ONCE somewhere else. But I have not figure out where yet? Am I being delusional again? 🙂
-Mike W.
@Mike W.: Agree some kind of a global listener would be simpler to use. But, this is how the APIs are implemented now. The stream listeners must be registered for each response. Perhaps you could file a new bug in bugzilla asking for better APIs (if possible)...
@Honza
Thanks for the reply. Firebug is pretty much the most helpful addon that I used. So I've been study its code in my spare time. Here is my thought. Since the setNewListener() is on the nsITraceableChannel interface. It seemed that we could register the listener at where we register the other observers. In the tabCache.js, initializeUI(),
// Register for HTTP events.
if (Ci.nsITraceableChannel)
httpObserver.addObserver(this, "firebug-http-event", false);
//?? can we call setNewLister() here??
still not clear on what's the difference between an observer and a streamLister yet. seems to me that they are all just kinds of "observers", just observing at a different granularities.
Lastly, where do you get more help in developing extensions, the document on developer.mozilla.org is a bit sketchy.
Hy,
I'm using a code like the one you have provided here. The code works fine while reading data, but the pages doesn't show up in the browser.
Anyone can help with an idea about what's happening?
My js is available at:
http://codeviewer.org/view/code:8e4
I have also to say that I'm running the code through a HTML file.
PS: Sorry the bad English 😉
@Hugo Dias: Reading inputStream (in onDataAvailable) moves the cursor to the end so, the original listener can't read anything. The solution is to copy entire the stream and pass it (with the cursor at the beginning) to the original listener. Perhaps this could be the problem...?
See also Firebug's implementation here: http://code.google.com/p/fbug/source/browse/branches/firebug1.4/components/firebug-channel-listener.js#80
Hi Honza,
Very nice article on this subject. I was trying these things and curious to know following thing,
onDataAvailable works only when http-on-examine-response event triggered. But i am unable get to this if the page is cached. I tried using http-on-examine-merged-response and http-on-examine-cached-response, but no use. I use FF 3.0.12, any suggestion?
Also is there any best source of documentation?
credit restoration...
This is a great site, I only wish I had found it sooner....
[...] following post draws significantly from a post by Jan Odvarko at http://www.softwareishard.com/blog/firebug/nsitraceablechannel-intercept-http-traffic/ but goes a bit further and addresses a bug in his code. There are also some sections which were [...]
Great article! I have created a note about this at the end of my post.
Thank You for this great article - it was really useful for me. I'm playing with this code for a while and I found a bug. There is:
var responseSource = this.receivedData.join();
And in a fact there should be
var responseSource = this.receivedData.join('');
Because the default separator for join is a coma ',' ! So if You experienced problems with big files - that's probably the bug 🙂
Hi Honza,
Thank you for this post and your help on firebug IRC. I was looking for a method to intercept form data for different sites in a firefox extension and by following your tutorial and Jonathan's I could do that.
However I could not intercept the form data on one site (http://blogger.com -> create new blog post) where request.requestMethod always returned GET on form submit although the form has a POST method. I noticed that Firebug captures the POST data correctly for this form though.
Greatly appreciate if you can give some pointers on how to solve this error.
Thank you!
Firebug uses readPostTextFromRequest and readPostTextFromPage (see lib.js. http://code.google.com/p/fbug/source/browse/branches/firebug1.6/content/firebug/lib.js) methods to get posted data (they are called from net.js).
Hi Honza,
Thank you for this great article!
Dou you have this example in a single file? I get an syntax error in this point:
...
throw Components.results.NS_NOINTERFACE;
}
}); <-- Error
Thanks!