Subscribe to RSS Feed

HTTP Tracing Export Format

by Honza

One of the new features planned for future version of Firebug is export of all collected data within the Net panel (HTTP tracing info). This feature has been requested several times and it should be useful mainly for further analysis by other tools.

There is currently several existing tools that can be used for HTTP tracking. To name a few:

They have various advantages over each other. For example, in-browser tools can easily group requests by page and analyze browser-cache usage while network-level tools can easily gather low level detailed info (e.g. HTTP compression). But in general, they all can be used to track HTTP traffic.

It would be obviously very beneficial to have a common export/import format that is used across all HTTP tracing tools and perhaps other projects. This would allow effective processing and analyzing data coming from various sources.

I have put together a document (fist draft) that represents a proposal for HTTP Archive export/import format (based on HTTPWatch's structure, but designed for JSON). Any comments and suggestions are greatly appreciated!

Many thanks to Simon Perkins (HTTPWatch) who provides all necessary info.

A viewer for HTTP Archive data is available here.
A NetExport extension for exporting data from the Net panel is available here.

Rss Commenti

25 Comments

  1. We were actually talking about adding more HTTP inspection features in to Windmill (http://www.getwindmill.com) today.

    Maybe we can add this as an export format so that you could log all the proxy data and then sort through it in an external application.

    #1 mikeal

  2. Yes, this would be great! If Windmill supports this format all logs could be also imported and analyzed using Firebug (the plan is to have this in Firebug 1.5).

    #2 admin

  3. [...] Jan Advarko of the Firebug team is working on just that: There is currently several existing tools that can be used for HTTP tracking. To name a [...]

    #3 Ajaxian » Common HTTP Tracing Export Format

  4. Very timmely, I've been working with a colleague in just this space. Charles (itself cross-platform and inherently cross-browser) has a nice choice of export formats, and it can re-import the same to let you quickly sort and summarize and waterfall-chart a session.
    We're writing out the same format from our own code (config-style property:value list) and importing into Charles. It would be really really nice to be able to do this once for all tools. And have open-source libraries available for both producing and consuming the same.

    #4 Sam Foster

  5. [...] imported it into Firebug, and I see what you mean.” Well, Jan Advarko of the Firebug team is working on just that:There is currently several existing tools that can be... more on the original website Report This [...]

    #5 linkfeedr » Blog Archive » Common HTTP Tracing Export Format - RSS Indexer (beta)

  6. @Sam: Yes this is exactly the idea. We are open to any suggestions that make the current proposal flexible and suitable for wide adoption. I am also planning to have JSON Schema (http://www.json.com/json-schema-proposal/) so, it's easy for any tool to validate before import. Please, let me know if you have any ideas!

    #6 admin

  7. Jan, have you take a look at "HTTP Vocabulary in RDF" (http://www.w3.org/TR/HTTP-in-RDF/) being defined at the W3C? I know that RDF is a bit of a pain to handle, but these guys are doing a great job a covering a lot of the interactions happening at the HTTP level. My 2 cents!

    #7 Rui Lopes

  8. Rockstarapps has just launch the RockStar Web Profiler that give people both standard proxy type information about resources as well as JavaScript execution timing.

    I think it would be good if we supported this as well.

    http://www.rockstarapps.com

    #8 Bob (Buffone)

  9. Looks good Jan. I noticed it seems to be missing a spec for POSTed files though. Would it just save the encoded content of the file (assuming binary)?

    #9 Dan

  10. I feel like you'll have a good answer to this, but why not use existing pcap formats? I know pcap formats can be used for much more then http tracing but it would already include the data that is needed. And honestly what if we wanted wanted firebug for other type of requests? (ie. ssh ftp sftp ftps streaming formats)

    #10 Francis

  11. @Rui: Yeah, I am also not big fan of RDF, but the HTTP Vocabulary is definitely something I want to keep in mind.

    @Francis: Yeah, my feeling is that this format does too much and would be heavy weight for HTTP Tracing. Also notice that the format is intended for HTTP and should target web page load performance analytic tools.
    Thanks for tips!

    #11 admin

  12. @Bob: yes, posted date should be encoded and saved. I'll have to update the doc about this yet.

    #12 admin

  13. Great plan. I had this in mind with my HTTP Trace format but never made any noise about it. I'll definitely look to support this JSON format ASAP.

    My HTTP Trace format is at a lower level than what you're proposing - it just outputs request and response data. It's an interesting idea to embed the information about pages into the export - it will make using the data less complex!

    #13 Karl von Randow

  14. So yeah, I didn't mention above - I'm the creator of Charles Web Debugging Proxy (which should be in your list above!) - so I will add this format into Charles and be in touch with any comments or when there's something to try!

    #14 Karl von Randow

  15. [...] Jan Odvarko of the Firebug team is working on just that: There is currently several existing tools that can be used for HTTP tracking. To name a [...]

    #15 Common HTTP Tracing Export Format | Guilda Blog

  16. @Karl: Great! Using the format in Charles would be another opportunity how to test its flexibility.
    Btw. the list is updated, Charles included now.

    #16 admin

  17. [...] Jan Odvarko of the Firebug team is working on just that: There is currently several existing tools that can be used for HTTP tracking. To name a [...]

    #17 Common HTTP Tracing Export Format

  18. I have to second Rui Lopes, but RDF`s are not a pain any more than XML`s can be.

    #18 Horia Dragomir

  19. it didn't install properly for me - using FF 3 on ubuntu hardy heron.. didn't give any errors but the toolbar / export button doesn't show up.

    #19 David Wilhelm

  20. @David Vilhelm: Could you please report a new issue here: http://code.google.com/p/fbug/issues/list
    and let's continue the discussion under the issue.
    Thanks!

    #20 Honza

  21. Great job, I wish I could automatically export Net panel infos to a file on the disk...

    #21 spirit

  22. @spirit: http://www.softwareishard.com/blog/firebug/exporting-http-tracing-data-from-firebug/

    #22 Honza

  23. well yes 🙂 but what I want is that when I open a new tab, then the Net stuff is exported on a file on my hard disk automatically so that I can process it later.

    #23 spirit

  24. @spirit: Ah, I see. Please report a bug here:
    http://code.google.com/p/fbug/issues/list
    and I'll do it.

    #24 Honza

  25. I added a ticket: http://code.google.com/p/fbug/issues/detail?id=2351

    #25 spirit

Sorry, the comment form is closed at this time.