Search This Blog

Labels

Showing posts with label blender. Show all posts
Showing posts with label blender. Show all posts

Sunday, December 26, 2010

The Generic Handy Operating System Toolkit

Introduction

The Generic Handy Operating System Toolkit (GHOST) is a windowing wrapper which was originally written at NaN Technologies BV to support their program Blender. GHOST was written when the implementation of GLUT on Mac OSX was shown to be inadequate, and since the GLUT sources were not available at that time, NaN chose to write their own GLUT replacement, thus GHOST was born.
With the open sourcing of Blender, the decision has been made to split the GHOST library out into its own project. This being done for maintenance purposes, and to allow others in the community to work with the GHOST library without having to bother with the Blender sources.

Because GHOST is a replacement for GLUT, it is able to handle all of the GLUT requirements which arise in Blender. These include:

  • Timer management.
  • Display / window management (only on the main display).
  • Event management.
  • Cursor shape management (currently no custom cursors).
  • Access to mouse buttons, mouse wheel, and keyboard information.

This article is going to show how to create a GHOST program, a simple rotating cube, which will accept keyboard and mouse wheel input.
This tutorial is going to assume that you have C coding experience and have used OpenGL before, as it will be used for the drawing. Even though this article uses C, GHOST is actually written in C++, but contains a C wrapper API. As I prefer C to C++ this article will be using that API. The concepts and functions will be similar if you wish to use C++, but you'll need to figure out the exact functions and objects to use.
Also, this program has only been complied on Linux. GHOST has support for Unix, Windows and Mac OS X systems built into it, but there is no guarentee that this example program will execute on any system other than Linux without modification.

Getting and installing GHOST

GHOST is currently only available through CVS. The repository can be accessed fromprojects.blender.org/projects/ghost. All the latest information on GHOST can be found on this site. If any bugs are found while developing they can be filed here under the tracker. The checkout directions are available under the CVS tab.
In order to compile GHOST successfully you will need a fairly recent version of autoconf/automake. When I compiled I needed to use:

  • autoconf 2.57
  • automake 1.6

Older versions may work, but then they may not. If you receive errors while doing a ./bootstrap then try upgrading your autoconf/automake.
To compile GHOST use the following commands (change /usr/local to where you wish the library to install to):

      ./bootstrap
./configure --prefix=/usr/local/
make
su
make install

If everything goes correctly you will have libGHOST.a installed under /usr/local/lib and will have the GHOST headers installed in /usr/local/include/GHOST.


The code

The example program listed here is a simple one that I used to learn the basics of GHOST. The program is pretty simple; it creates the necessary GHOST objects to show a window, draws an OpenGL box inside the window, then adds a timer when the 't' key is pressed to rotate the box. The mouse wheel is used to rotate a small amount in the X-direction whenever it is rolled.
The code listings in the article will only give the GHOST relevant portions of the code. The full code listing is availablehere.



  #include <stdlib.h>
#include <stdio.h>
#include <GHOST_C-api.h>
#include <GL/gl.h>

We will be using the C API as mentioned above, and most all GHOST applications will require the GL/gl.h library to do the OpenGL work.



  static GHOST_SystemHandle ghost_system;
static GHOST_EventConsumerHandle consumer;
static GHOST_WindowHandle win;
static GHOST_WindowHandle full_screen_window;
static GHOST_TimerTaskHandle timer;

Next we create all of our global variables to store GHOST information. The first line creates the system. All GHOST apps are required to have one (and only one) GHOST_SystemHandle. This handle will be passed to many of the GHOST functions used throughout the program.
The second line is for the GHOST_EventConsumerHandle, this handle will have a function pointer attached to it on creation which will be called whenever events arrive through the system.
Third and fourth lines are the windows which will be used in the application, the first is the main window, and the second is used when we move into full screen mode.
The fifth line is the declaration of a GHOST timer which will be used when we want to rotate our box.



  void invalidate_window(void) {
if (GHOST_GetFullScreen(ghost_system))
GHOST_InvalidateWindow(full_screen_window);
else {
if (GHOST_ValidWindow(ghost_system, win))
GHOST_InvalidateWindow(win);
}
}

The invalidate_window routine will be used when we need to tell GHOST that we have modified the window contents. This will cause GHOST to generate a GHOST_kWindowUpdate event which we will be handled later.



  void timer_proc(GHOST_TimerTaskHandle task, GHOST_TUns64 time) {
update_window();
invalidate_window();
}

This is the procedure which we will attach to the timer when it is created. It will be called whenever the timer event goes off. The parameters to the function are: a handle to the timer which caused the function call, and the time at which the timer went off.



  void setup_window(GHOST_WindowHandle win) {
GHOST_RectangleHandle rect = NULL;
GLfloat w, h, aspect;

GHOST_ActivateWindowDrawingContext(win);
rect = GHOST_GetClientBounds(win);

w = (GLfloat)GHOST_GetWidthRectangle(rect);
h = (GLfloat)GHOST_GetHeightRectangle(rect);
GHOST_DisposeRectangle(rect);
...
}

The setup_window routine is doing the initial setup of the GHOST window, through the call to GHOST_ActivateWindowDrawingContext and then receives the size of the window through the GHOST_GetClientBounds. We can then get the width and height of the window to be used in the OpenGL initialization routines.



  int process_event(GHOST_EventHandle event, GHOST_TUserDataPtr data) {

The process_event procedure will be attached to the event consumer when it is created. When an event happens the routines attached as consumers will be called and passed the event which caused the call, and any data which has been attached to the call-back.




int handled = 0;
GHOST_TEventKeyData *key_data =
(GHOST_TEventKeyData *)GHOST_GetEventData(event);

The GHOST_GetEventData call will return any keyboard data attached to this event.




switch (GHOST_GetEventType(event)) {
case GHOST_kEventUnknown:
case GHOST_kEventCursorMove:
case GHOST_kEventButtonDown:
case GHOST_kEventButtonUp:
case GHOST_kEventKeyUp:
case GHOST_kEventWindowActivate:
case GHOST_kEventWindowDeactivate:
case GHOST_kEventWindowSize:
case GHOST_kNumEventTypes:
break;

case GHOST_kEventWheel:
{
GHOST_TEventWheelData *wheel =
(GHOST_TEventWheelData *)GHOST_GetEventData(event);

if (wheel->z > 0)
xrot += 2;
else
xrot -= 2;

update_window();
invalidate_window();
}
break;

If you have a mouse wheel, GHOST is able to handle the movement of the wheel. Movement in one direction returns a wheel->z of -1 and in the other direction of 1. So, action can be taken as appropriate for the wheel motion.




case GHOST_kEventWindowUpdate:
{
GHOST_WindowHandle win_handle =
GHOST_GetEventWindow(event);
if (!GHOST_ValidWindow(ghost_system, win_handle))
break;

setup_window(win_handle);
update_window();
GHOST_SwapWindowBuffers(win_handle);
}
break;

A GHOST_kEventWindowUpdate event will be received whenever GHOST needs the window to be updated. So we initialize the OpenGL context with the setup_window call, update our content and tell GHOST to swap the drawing buffers with GHOST_SwapWindowBuffers. The GHOST_SwapWindowBuffers function requires the window to be passed. The window we are currently using can retrieved from the event data with the GHOST_GetEventWindow function call.




case GHOST_kEventQuit:
case GHOST_kEventWindowClose:
exit_requested = 1;
handled = 1;
break;

case GHOST_kEventKeyDown:
if (key_data) {
if (key_data->key == GHOST_kKeyQ) {
exit_requested = 1;
handled = 1;
} else if (key_data->key == GHOST_kKeyR) {
xrot = 0;
yrot = 0;
zrot = 0;

invalidate_window();

The GHOST_kEventKeyDown is generated whenever a key on the key board is pressed. From the key_data gathered above we can then determine the key that has been pressed and act accordingly.




} else if (key_data->key == GHOST_kKeyT) {
if (!timer)
timer =
GHOST_InstallTimer(ghost_system, 0, 10,
timer_proc, NULL);
else {
GHOST_RemoveTimer(ghost_system, timer);
timer = NULL;
}


Timers can be added and removed from the system through the GHOST_InstallTimer and GHOST_RemoveTimer function calls. The GHOST_InstallTimer call is passed the system, the delay before starting the timer, the interval between calls to the timer, the function to call when the timer expires, and any data to pass to the timer function.




} else if (key_data->key == GHOST_kKeyF) {
if (GHOST_GetFullScreen(ghost_system)) {
GHOST_EndFullScreen(ghost_system);
full_screen_window = NULL;
} else {
GHOST_DisplaySetting setting;
setting.bpp = 24;
setting.frequency = 85;
setting.xPixels = 640;
setting.yPixels = 480;

full_screen_window =
GHOST_BeginFullScreen(ghost_system,
&setting, 0);
}


GHOST is able to handle full screen as well as regular window mode. It is possible to determine if the system is running in full screen or not through the GHOST_GetFullScreen function call. This will return true if the system is running full screen.
If the system is in full screen mode, we can switch back to normal mode with the GHOST_EndFullScreen function call. This will remove the full screen window and you will be back to the size you were at before entering full screen mode.
To enter into full screen mode you need to create a GHOST_DisplaySetting object which is then passed to the GHOST_BeginFullScreen function call. The last parameter to GHOST_BeginFullScreen is a boolean stating whether to set the system into stereo vision mode. The DisplaySettings is given the parameters needed to setup the system to execute in full screen mode. They are:




  • bpp - Number of bits per pixel.


  • frequency - The refresh rate (in Hertz).


  • xPixels - The number of pixels on a line.


  • yPixels - The number of lines.

The full screen mode produced by GHOST actually produces another window which runs at full screen. If you minimize the full screen window you will be able to see your original window still running in the background.



  int main(int argc, char ** argv) {
ghost_system = GHOST_CreateSystem();
if (!ghost_system) {
printf("Coun't not create the system, dying\n");
exit(-1);
}

The first thing we do is to create the system which is to be used throughout the program to call GHOST functions. If we can't get it then there is no reason to continue. The system needs to be created before any other GHOST functions can be called.




consumer = GHOST_CreateEventConsumer(process_event, NULL);
if (!consumer) {
printf("Failed to create consumer\n");
GHOST_DisposeSystem(ghost_system);
exit(-1);
}
GHOST_AddEventConsumer(ghost_system, consumer);

Once we have the system, we can create an event consumer. The GHOST_CreateEventConsumer takes the function pointer of the function to call when an event happens and any user data to pass to the given function. If we can't get the consumer, again, there is no real reason to continue. Once the consumer is created we attach it to the system as an event consumer.




win = GHOST_CreateWindow(ghost_system, "cube-or", 10, 64, 320, 200,
GHOST_kWindowStateNormal, GHOST_kDrawingContextTypeOpenGL, 0);
if (!win) {
printf("Couldn't not create window\n");
GHOST_DisposeSystem(ghost_system);
GHOST_DisposeEventConsumer(consumer);
exit(-1);
}

We can now continue on and create the application window. This is done through a call to GHOST_CreateWindow. The parameters are as follows:




  • The system.


  • The title.


  • The position of the left edge of the window.


  • The position of the top of the window.


  • The width of the window.


  • The height of the window.


  • The state of the window.


  • The drawing context the window will be in.


  • If the window is in stereo visual mode.

The state of the window is of type GHOST_TWindowState and can be one of: GHOST_kWindowStateNormal, GHOST_kWindowStateMinimized, GHOST_kWindowStateMaximized, and GHOST_kWindowStateFullScreen. All of which should be pretty self explanatory.
The drawing context is of type GHOST_TDrawingContextType which is either: GHOST_kDrawingContextTypeNone, or GHOST_kDrawingContextTypeOpenGL. So you may as well set it to the OpenGL context if you want anything useful.




while(!exit_requested) {
GHOST_ProcessEvents(ghost_system, 0);
GHOST_DispatchEvents(ghost_system);
GHOST_SwapWindowBuffers(win);
}

This is the main loop of the program. It adds events to the event queue in the GHOST_ProcessEvents call, and then dispatches those events to the event consumers in the GHOST_DispatchEvents call. The GHOST_ProcessEvents function receives the system and a boolean which determines if the system blocks until the next event is received. The GHOST_ProcessEvents call will actually return a variable relating the success it had waiting for events (which could be caught to have the program sleep if there are no events to process.)




if (GHOST_GetFullScreen(ghost_system))
GHOST_EndFullScreen(ghost_system);

if (timer)
GHOST_RemoveTimer(ghost_system, timer);

if (GHOST_ValidWindow(ghost_system, win))
GHOST_DisposeWindow(ghost_system, win);

GHOST_DisposeSystem(ghost_system);
GHOST_DisposeEventConsumer(consumer);
return 0;
}

When the main loop is finished we need to clean up after ourselves. This includes moving out of full screen mode, deleting the timer if it exists, deleting the window if it is still valid, cleaning up the system and cleaning up any consumers.
And that's it. Those are the guts of a GHOST application (missing a few unimportant bits). The only step left is to compile the code. There is a ghost-config script under development but it doesn't quite work correctly for libraries at the time of this articles writing, so I'll just give you the compile command I use for the above program.



gcc -I/usr/local/include -I/usr/local/include/GHOST -g \
-O2 -Wall -funsigned-char -L /usr/X11R6/lib -o cube-or \
main.c -L /usr/local/lib/libGHOST.a -lGLU \
-lGL -lXxf86dga -lX11 -lXext -lutil -lm -lpthread \
-ldl -lstdc++ -lz -lGHOST

Of course any paths will need to be adjusted for your local machine. As a note, GHOST will spit out debug information on mouse movements and key presses while it is running, this is normal and to be expected.
That should be enough information for you to get some GHOST development off the ground. There are lots of routines left in the library that were not touched upon by this article and are waiting for someone to discover them.


Contributing to GHOST

If after all of this your interested in contributing to the GHOST project feel free to check out the projects site listed above and get in contact with the developers. There is currently a need for more example / test programs and for people to write API documentation. If you find bugs, have feature requests or patches, you can submit them on the project site given above.

References

The introduction section on the history of GHOST is from the GHOST documentation.
The example given here draws heavily on the monkey example in the current CVS. Without which there would be no example.


About

This article was written by dan sinclair (dj2 on #blendersauce). If you wish to contact me about this article send an email to zero at everburning dot com.

Monday, December 20, 2010

The mystery of the blend——The blender file-format explained

The mystery of the blend
The blender file-format explained
Jeroen Bakker
j.bakker@atmind.nl
http://www.atmind.nl/blender
06-10-2010

Introduction

In this article I will describe the blend-file-format with a request to tool-makers to support blend-file.

First I'll describe how Blender works with blend-files. You'll notice why the blend-file-format is not that well documented, as from Blender's perspective this is not needed. We look at the global file-structure of a blend-file (the file-header and file-blocks). After this is explained, we go deeper to the core of the blend-file, the DNA-structures. They hold the blue-prints of the blend-file and the key asset of understanding blend-files. When that's done we can use these DNA-structures to read information from elsewhere in the blend-file.

In this article we'll be using the default blend-file from Blender 2.54, with the goal to read the output resolution from the Scene. The article is written to be programming language independent and I've setup a web-site for support.

Loading and saving in Blender

Loading and saving in Blender is very fast and Blender is known to have excellent downward and upward compatibility. Ton Roosendaal demonstrated that in December 2008 by loading a 1.0 blend-file using Blender 2.48a
[ref: http://www.blendernation.com/2008/12/01/blender-dna-rna-and-backward-compatibility/].

Saving complex scenes in Blender is done within seconds. Blender achieves this by saving data in memory to disk without any transformations or translations. Blender only adds file-block-headers to this data. A file-block-header contains clues on how to interpret the data. After the data, all internally Blender structures are stored. These structures will act as blue-prints when Blender loads the file. Blend-files can be different when stored on different hardware platforms or Blender releases. There is no effort taken to make blend-files binary the same. Blender creates the blend-files in this manner since release 1.0. Backward and upwards compatibility is not implemented when saving the file, this is done during loading.

When Blender loads a blend-file, the DNA-structures are read first. Blender creates a catalog of these DNA-structures. Blender uses this catalog together with the data in the file, the internal Blender structures of the Blender release you're using and a lot of transformation and translation logic to implement the backward and upward compatibility. In the source code of blender there is actually logic which can transform and translate every structure used by a Blender release to the one of the release you're using
[ref: http://download.blender.org/source/blender-2.48a.tar.gz blender/blenloader/intern/readfile.c lines 4946-7960].
The more difference between releases the more logic is executed.

The blend-file-format is not well documented, as it does not differ from internally used structures and the file can really explain itself.

Global file-structure

This section explains how the global file-structure can be read.

  • A blend-file always start with the file-header
  • After the file-header, follows a list of file-blocks (the default blend file of Blender 2.48 contains more than 400 of these file-blocks).
  • Each file-block has a file-block header and file-block data
  • At the end of the blend-file there is a section called "Structure DNA", which lists all the internal structures of the Blender release the file was created in
  • The blend-file ends with a file-block called 'ENDB'

File.blend

File-header

File-block

Header

Data

File-block

File-block

File-block 'Structure DNA'

Header ('DNA1')

Data ('SDNA')

Names ('NAME')

Types ('TYPE')

Lengths ('TLEN')

Structures ('STRC')

File-Block 'ENDB'

File-Header

The first 12 bytes of every blend-file is the file-header. The file-header has information on Blender (version-number) and the PC the blend-file was saved on (pointer-size and endianness). This is required as all data inside the blend-file is ordered in that way, because no translation or transformation is done during saving. The next table describes the information in the file-header.

File-header
reference structure type offset size
identifier char[7] File identifier (always 'BLENDER') 0 7
pointer-size char Size of a pointer; all pointers in the file are stored in this format. '_' means 4 bytes or 32 bit and '-' means 8 bytes or 64 bits. 7 1
endianness char Type of byte ordering used; 'v' means little endian and 'V' means big endian. 8 1
version-number char[3] Version of Blender the file was created in; '254' means version 2.54 9 3

Endianness addresses the way values are ordered in a sequence of bytes(see the example below):

  • in a big endian ordering, the largest part of the value is placed on the first byte and the lowest part of the value is placed on the last byte,
  • in a little endian ordering, largest part of the value is placed on the last byte and the smallest part of the value is placed on the first byte.

Nowadays, little-endian is the most commonly used.

Endianess Example

Writing the integer 0x4A3B2C1Dh, will be ordered:

  • in big endian as 0x4Ah, 0x3Bh, 0x2Ch, 0x1Dh
  • in little endian as 0x1Dh, 0x2Ch, 0x3Bh, 0x4Ah

Blender supports little-endian and big-endian.
This means that when the endianness is different between the blend-file and the PC your using, Blender changes it to the byte ordering of your PC.

File-header Example

This hex-dump describes a file-header created with blender 2.54.0 on little-endian hardware with a 32 bits pointer length.

pointer-size version-number | | 0000 0000: [42 4C 45 4E 44 45 52] [5F] [76] [32 35 34] BLENDER_v254 | | identifier endianness

File-blocks

File-blocks contain a "file-block header" and "file-block data".

File-block headers

The file-block-header describes:

  • the type of information stored in the file-block
  • the total length of the data
  • the old memory pointer at the moment the data was written to disk
  • the number of items of this information

As we can see below, depending on the pointer-size stored in the file-header, a file-block-header can be 20 or 24 bytes long, hence it is always aligned at 4 bytes.

File-block-header
reference structure type offset size
code char[4] File-block identifier 0 4
size integer Total length of the data after the file-block-header 4 4
old memory address void* Memory address the structure was located when written to disk 8 pointer-size (4/8)
SDNA index integer Index of the SDNA structure 8+pointer-size 4
count integer Number of structure located in this file-block 12+pointer-size 4

The above table describes how a file-block-header is structured:

  • Code describes different types of file-blocks. The code determines with what logic the data must be read.
    These codes also allows fast finding of data like Library, Scenes, Object or Materials as they all have a specific code.
  • Size contains the total length of data after the file-block-header. After the data a new file-block starts. The last file-block in the file has code 'ENDB'.
  • Old memory address contains the memory address when the structure was last stored. When loading the file the structures can be placed on different memory addresses. Blender updates pointers to these structures to the new memory addresses.
  • SDNA index contains the index in the DNA structures to be used when reading this file-block-data.
    More information about this subject will be explained in the Reading scene information section.
  • Count tells how many elements of the specific SDNA structure can be found in the data.

Example

This hex-dump describes a File-block (= File-block header + File-block data) created with blender 2.54 on little-endian hardware with a 32 bits pointer length.

file-block identifier='SC' data size=1404 old pointer SDNA index=150 | | | | 0000 4420: [53 43 00 00] [7C 05 00 00] [68 34 FB 0B] [96 00 00 00] SC.. `... ./.. .... 0000 4430: [01 00 00 00] [xx xx xx xx xx xx xx xx xx xx xx xx .... xxxx xxxx xxxx | | count=1 file-block data (next 1404 bytes)
  • The code 'SC'+0x00h identifies that it is a Scene.
  • Size of the data is 1404 bytes (0x0000057Ch = 0x7Ch + 0x05h * 256 = 124 + 1280)
  • The old pointer is 0x0BFB3468h
  • The SDNA index is 150 (0x00000096h = 6 + 9 * 16 = 6 + 144)
  • The section contains a single scene (count = 1).

Before we can interpret the data of this file-block we first have to read the DNA structures in the file. The section "Structure DNA" will show how to do that.

Structure DNA

The DNA1 file-block

Structure DNA is stored in a file-block with code 'DNA1'. It can be just before the 'ENDB' file-block.

The 'DNA1' file-block contains all internal structures of the Blender release the file was created in.
These structure can be described as C-structures: they can hold fields, arrays and pointers to other structures, just like a normal C-structure.

struct SceneRenderLayer { struct SceneRenderLayer *next, *prev; char name[32]; struct Material *mat_override; struct Group *light_override; unsigned int lay; unsigned int lay_zmask; int layflag; int pad; int passflag; int pass_xor; };

For example,a blend-file created with Blender 2.54 the 'DNA1' file-block is 57796 bytes long and contains 398 structures.

DNA1 file-block-header

The DNA1 file-block header follows the same rules of any other file-block, see the example below.

Example

This hex-dump describes the file-block 'DNA1' header created with blender 2.54.0 on little-endian hardware with a 32 bits pointer length.

(file-block identifier='DNA1') data size=57796 old pointer SDNA index=0 | | | | 0004 B060 [44 4E 41 31] [C4 E1 00 00] [C8 00 84 0B] [00 00 00 00] DNA1............ 0004 B070 [01 00 00 00] [53 44 4E 41 4E 41 4D 45 CB 0B 00 00 ....SDNANAME.... | | count=1 'DNA1' file-block data (next 57796 bytes)

DNA1 file-block data

The next section describes how this information is ordered in the data of the 'DNA1' file-block.

Structure of the DNA file-block-data
repeat condition name type length description
identifier char[4] 4 'SDNA'
name identifier char[4] 4 'NAME'
#names integer 4 Number of names follows
for(#names) name char[] ? Zero terminating string of name, also contains pointer and simple array definitions (e.g. '*vertex[3]\0')
type identifier char[4] 4 'TYPE' this field is aligned at 4 bytes
#types integer 4 Number of types follows
for(#types) type char[] ? Zero terminating string of type (e.g. 'int\0')
length identifier char[4] 4 'TLEN' this field is aligned at 4 bytes
for(#types) length short 2 Length in bytes of type (e.g. 4)
structure identifier char[4] 4 'STRC' this field is aligned at 4 bytes
#structures integer 4 Number of structures follows
for(#structures) structure type short 2 Index in types containing the name of the structure
.. #fields short 2 Number of fields in this structure
.. for(#field) field type short 2 Index in type
for end for end field name short 2 Index in name

As you can see, the structures are stored in 4 arrays: names, types, lengths and structures. Every structure also contains an array of fields. A field is the combination of a type and a name. From this information a catalog of all structures can be constructed. The names are stored as how a C-developer defines them. This means that the name also defines pointers and arrays. (When a name starts with '*' it is used as a pointer. when the name contains for example '[3]' it is used as a array of 3 long.) In the types you'll find simple-types (like: 'integer', 'char', 'float'), but also complex-types like 'Scene' and 'MetaBall'. 'TLEN' part describes the length of the types. A 'char' is 1 byte, an 'integer' is 4 bytes and a 'Scene' is 1376 bytes long.

Note

All identifiers, are arrays of 4 chars, hence they are all aligned at 4 bytes.

Example

Created with blender 2.54.0 on little-endian hardware with a 32 bits pointer length.

The names array

The first names are: *next, *prev, *data, *first, *last, x, y, xmin, xmax, ymin, ymax, *pointer, group, val, val2, type, subtype, flag, name[32], ...

file-block-data identifier='SDNA' array-id='NAME' number of names=3019 | | | 0004 B070 01 00 00 00 [53 44 4E 41][4E 41 4D 45] [CB 0B 00 00] ....SDNANAME.... 0004 B080 [2A 6E 65 78 74 00][2A 70 72 65 76 00] [2A 64 61 74 *next.*prev.*dat | | | '*next\0' '*prev\0' '*dat' .... .... (3019 names)

Note

While reading the DNA you'll will come across some strange names like '(*doit)()'. These are method pointers and Blender updates them to the correct methods.

The types array

The first types are: char, uchar, short, ushort, int, long, ulong, float, double, void, Link, LinkData, ListBase, vec2s, vec2f, ...

array-id='TYPE' | 0005 2440 6F 6C 64 5B 34 5D 5B 34 5D 00 00 00 [54 59 50 45] old[4][4]...TYPE 0005 2450 [C9 01 00 00] [63 68 61 72 00] [75 63 68 61 72 00][73 ....char.uchar.s | | | | number of types=457 'char\0' 'uchar\0' 's' .... .... (457 types)

The lengths array

char uchar ushort short array-id length length length length 'TLEN' 1 1 2 2 0005 3AA0 45 00 00 00 [54 4C 45 4E] [01 00] [01 00] [02 00] [02 00] E...TLEN........ .... 0005 3AC0 [08 00] [04 00] [08 00] [10 00] [10 00] [14 00] [4C 00] [34 00] ............L.4. 8 4 8 ListBase vec2s vec2f ... etc length len length .... .... (457 lengths, same as number of types)

The structures array

array-id='STRC' | 0005 3E30 40 00 38 00 60 00 00 00 00 00 00 00 [53 54 52 43] @.8.`.......STRC 0005 3E40 [8E 01 00 00] [0A 00] [02 00] [0A 00] [00 00] [0A 00] [01 00] ................ 398 10 2 10 0 10 0 number of index fields index index index index structures in types in types in names in types in names ' '----------------' '-----------------' ' ' field 0 field 1 ' '--------------------------------------------------------' structure 0 .... .... (398 structures, each one describeing own type, and type/name for each field)

The DNA structures inside a Blender 2.48 blend-file can be found at http://www.atmind.nl/blender/blender-sdna.html. If we understand the DNA part of the file it is now possible to read information from other parts file-blocks. The next section will tell us how.

Reading scene information

Let us look at the file-block header we have seen earlier:

  • the file-block identifier is 'SC'+0x00h
  • the SDNA index is 150
  • the file-block size is 1404 bytes

Now note that:

  • the structure at index 150 in the DNA is a structure of type 'Scene' (counting from 0).
  • the associated type ('Scene') in the DNA has the length of 1404 bytes.

We can map the Scene structure on the data of the file-blocks. But before we can do that, we have to flatten the Scene-structure.

struct Scene { ID id; // 52 bytes long (ID is different a structure) AnimData *adt; // 4 bytes long (pointer to an AnimData structure) Object *camera; // 4 bytes long (pointer to an Object structure) World *world; // 4 bytes long (pointer to an Object structure) ... float cursor[3]; // 12 bytes long (array of 3 floats) ... };

The first field in the Scene-structure is of type 'ID' with the name 'id'. Inside the list of DNA structures there is a structure defined for type 'ID' (structure index 17).

struct ID { void *next, *prev; struct ID *newid; struct Library *lib; char name[24]; short us; short flag; int icon_id; IDProperty *properties; };

The first field in this structure has type 'void' and name '*next'.
Looking in the structure list there is no structure defined for type 'void': it is a simple type and therefore the data should be read. The name '*next' describes a pointer. As we see, the first 4 bytes of the data can be mapped to 'id.next'.

Using this method we'll map a structure to its data. If we want to read a specific field we know at which offset in the data it is located and how much space it takes.
The next table shows the output of this flattening process for some parts of the Scene-structure. Not all rows are described in the table as there is a lot of information in a Scene-structure.

Flattened SDNA structure 150: Scene
reference structure type name offset size description
id.next ID void *next 0 4 Refers to the next scene
id.prev ID void *prev 4 4 Refers to the previous scene
id.newid ID ID *newid 8 4
id.lib ID Library *lib 12 4
id.name ID char name[24] 16 24 'SC'+the name of the scene as displayed in Blender
id.us ID short us 40 2
id.flag ID short flag 42 2
id.icon_id ID int icon_id 44 4
id.properties ID IDProperty *properties 48 4
adt Scene AnimData *adt 52 4
camera Scene Object *camera 56 4 Pointer to the current camera
world Scene World *world 60 4 Pointer to the current world
Skipped rows
r.xsch RenderData short xsch 382 2 X-resolution of the output when rendered at 100%
r.ysch RenderData short ysch 384 2 Y-resolution of the output when rendered at 100%
r.xparts RenderData short xparts 386 2 Number of x-part used by the renderer
r.yparts RenderData short yparts 388 2 Number of x-part used by the renderer
Skipped rows
gpd Scene bGPdata *gpd 1376 4
physics_settings.gravity PhysicsSettings float gravity[3] 1380 12
physics_settings.flag PhysicsSettings int flag 1392 4
physics_settings.quick_cache_step PhysicsSettings int quick_cache_step 1396 4
physics_settings.rt PhysicsSettings int rt 1400 4

We can now read the X and Y resolution of the Scene:

  • the X-resolution is located on offset 382 of the file-block-data and must be read as a short.
  • the Y-resolution is located on offset 384 and is also a short

Note

An array of chars can mean 2 things. The field contains readable text or it contains an array of flags (not humanly readable).

Note

A file-block containing a list refers to the DNA structure and has a count larger than 1. For example Vertexes and Faces are stored in this way.

The mystery of the blend----The blender file-format explained


Jeroen Bakker






06-10-2010

Introduction


In this article I will describe the blend-file-format with a request to tool-makers to support blend-file.
First I'll describe how Blender works with blend-files. You'll notice why the blend-file-format is not that well documented, as from Blender's perspective this is not needed. We look at the global file-structure of a blend-file (the file-header and file-blocks). After this is explained, we go deeper to the core of the blend-file, the DNA-structures. They hold the blue-prints of the blend-file and the key asset of understanding blend-files. When that's done we can use these DNA-structures to read information from elsewhere in the blend-file.
In this article we'll be using the default blend-file from Blender 2.54, with the goal to read the output resolution from the Scene. The article is written to be programming language independent and I've setup a web-site for support.

Loading and saving in Blender


Loading and saving in Blender is very fast and Blender is known to have excellent downward and upward compatibility. Ton Roosendaal demonstrated that in December 2008 by loading a 1.0 blend-file using Blender 2.48a [ref: http://www.blendernation.com/2008/12/01/blender-dna-rna-and-backward-compatibility/].
Saving complex scenes in Blender is done within seconds. Blender achieves this by saving data in memory to disk without any transformations or translations. Blender only adds file-block-headers to this data. A file-block-header contains clues on how to interpret the data. After the data, all internally Blender structures are stored. These structures will act as blue-prints when Blender loads the file. Blend-files can be different when stored on different hardware platforms or Blender releases. There is no effort taken to make blend-files binary the same. Blender creates the blend-files in this manner since release 1.0. Backward and upwards compatibility is not implemented when saving the file, this is done during loading.
When Blender loads a blend-file, the DNA-structures are read first. Blender creates a catalog of these DNA-structures. Blender uses this catalog together with the data in the file, the internal Blender structures of the Blender release you're using and a lot of transformation and translation logic to implement the backward and upward compatibility. In the source code of blender there is actually logic which can transform and translate every structure used by a Blender release to the one of the release you're using [ref: http://download.blender.org/source/blender-2.48a.tar.gz blender/blenloader/intern/readfile.c lines 4946-7960]. The more difference between releases the more logic is executed.
The blend-file-format is not well documented, as it does not differ from internally used structures and the file can really explain itself.

Global file-structure


This section explains how the global file-structure can be read.
  • A blend-file always start with the file-header
  • After the file-header, follows a list of file-blocks (the default blend file of Blender 2.48 contains more than 400 of these file-blocks).
  • Each file-block has a file-block header and file-block data
  • At the end of the blend-file there is a section called "Structure DNA", which lists all the internal structures of the Blender release the file was created in
  • The blend-file ends with a file-block called 'ENDB'
File.blend
File-header
File-block
Header
Data
File-block
File-block
File-block 'Structure DNA'
Header ('DNA1')
Data ('SDNA')
Names ('NAME')
Types ('TYPE')
Lengths ('TLEN')
Structures ('STRC')
File-Block 'ENDB'

File-Header


The first 12 bytes of every blend-file is the file-header. The file-header has information on Blender (version-number) and the PC the blend-file was saved on (pointer-size and endianness). This is required as all data inside the blend-file is ordered in that way, because no translation or transformation is done during saving. The next table describes the information in the file-header.
File-header
referencestructuretypeoffsetsize
identifierchar[7]File identifier (always 'BLENDER')07
pointer-sizecharSize of a pointer; all pointers in the file are stored in this format. '_' means 4 bytes or 32 bit and '-' means 8 bytes or 64 bits.71
endiannesscharType of byte ordering used; 'v' means little endian and 'V' means big endian.81
version-numberchar[3]Version of Blender the file was created in; '254' means version 2.5493
Endianness addresses the way values are ordered in a sequence of bytes(see the example below):
  • in a big endian ordering, the largest part of the value is placed on the first byte and the lowest part of the value is placed on the last byte,
  • in a little endian ordering, largest part of the value is placed on the last byte and the smallest part of the value is placed on the first byte.
Nowadays, little-endian is the most commonly used.

Endianess Example
Writing the integer 0x4A3B2C1Dh, will be ordered:
  • in big endian as 0x4Ah0x3Bh0x2Ch0x1Dh
  • in little endian as 0x1Dh0x2Ch0x3Bh0x4Ah
Blender supports little-endian and big-endian.
This means that when the endianness is different between the blend-file and the PC your using, Blender changes it to the byte ordering of your PC.

File-header Example

This hex-dump describes a file-header created with blender 2.54.0 on little-endian hardware with a 32 bitspointer length.
pointer-size version-number | | 0000 0000: [42 4C 45 4E 44 45 52] [5F] [76] [32 35 34] BLENDER_v254 | | identifier endianness

File-blocks


File-blocks contain a "file-block header" and "file-block data".

File-block headers


The file-block-header describes:
  • the type of information stored in the file-block
  • the total length of the data
  • the old memory pointer at the moment the data was written to disk
  • the number of items of this information
As we can see below, depending on the pointer-size stored in the file-header, a file-block-header can be 20 or 24 bytes long, hence it is always aligned at 4 bytes.
File-block-header
referencestructuretypeoffsetsize
codechar[4]File-block identifier04
sizeintegerTotal length of the data after the file-block-header44
old memory addressvoid*Memory address the structure was located when written to disk8pointer-size (4/8)
SDNA indexintegerIndex of the SDNA structure8+pointer-size4
countintegerNumber of structure located in this file-block12+pointer-size4
The above table describes how a file-block-header is structured:
  • Code describes different types of file-blocks. The code determines with what logic the data must be read.
    These codes also allows fast finding of data like Library, Scenes, Object or Materials as they all have a specific code.
  • Size contains the total length of data after the file-block-header. After the data a new file-block starts. The last file-block in the file has code 'ENDB'.
  • Old memory address contains the memory address when the structure was last stored. When loading the file the structures can be placed on different memory addresses. Blender updates pointers to these structures to the new memory addresses.
  • SDNA index contains the index in the DNA structures to be used when reading this file-block-data.
    More information about this subject will be explained in the Reading scene information section.
  • Count tells how many elements of the specific SDNA structure can be found in the data.

Example
This hex-dump describes a File-block (= File-block header + File-block data) created with blender 2.54 onlittle-endian hardware with a 32 bits pointer length.
file-block identifier='SC' data size=1404 old pointer SDNA index=150 | | | | 0000 4420: [53 43 00 00] [7C 05 00 00] [68 34 FB 0B] [96 00 00 00] SC.. `... ./.. .... 0000 4430: [01 00 00 00] [xx xx xx xx xx xx xx xx xx xx xx xx .... xxxx xxxx xxxx | | count=1 file-block data (next 1404 bytes)
  • The code 'SC'+0x00h identifies that it is a Scene.
  • Size of the data is 1404 bytes (0x0000057Ch = 0x7Ch + 0x05h * 256 = 124 + 1280)
  • The old pointer is 0x0BFB3468h
  • The SDNA index is 150 (0x00000096h = 6 + 9 * 16 = 6 + 144)
  • The section contains a single scene (count = 1).
Before we can interpret the data of this file-block we first have to read the DNA structures in the file. The section "Structure DNA" will show how to do that.

Structure DNA


The DNA1 file-block


Structure DNA is stored in a file-block with code 'DNA1'. It can be just before the 'ENDB' file-block.
The 'DNA1' file-block contains all internal structures of the Blender release the file was created in.
These structure can be described as C-structures: they can hold fields, arrays and pointers to other structures, just like a normal C-structure.
struct SceneRenderLayer { struct SceneRenderLayer *next, *prev; char name[32]; struct Material *mat_override; struct Group *light_override; unsigned int lay; unsigned int lay_zmask; int layflag; int pad; int passflag; int pass_xor; };
For example,a blend-file created with Blender 2.54 the 'DNA1' file-block is 57796 bytes long and contains 398 structures.

DNA1 file-block-header


The DNA1 file-block header follows the same rules of any other file-block, see the example below.

Example
This hex-dump describes the file-block 'DNA1' header created with blender 2.54.0 on little-endian hardware with a 32 bits pointer length.
(file-block identifier='DNA1') data size=57796 old pointer SDNA index=0 | | | | 0004 B060 [44 4E 41 31] [C4 E1 00 00] [C8 00 84 0B] [00 00 00 00] DNA1............ 0004 B070 [01 00 00 00] [53 44 4E 41 4E 41 4D 45 CB 0B 00 00 ....SDNANAME.... | | count=1 'DNA1' file-block data (next 57796 bytes)

DNA1 file-block data


The next section describes how this information is ordered in the data of the 'DNA1' file-block.
Structure of the DNA file-block-data
repeat conditionnametypelengthdescription
identifierchar[4]4'SDNA'
name identifierchar[4]4'NAME'
#namesinteger4Number of names follows
for(#names)namechar[]?Zero terminating string of name, also contains pointer and simple array definitions (e.g. '*vertex[3]\0')
type identifierchar[4]4'TYPE' this field is aligned at 4 bytes
#typesinteger4Number of types follows
for(#types)typechar[]?Zero terminating string of type (e.g. 'int\0')
length identifierchar[4]4'TLEN' this field is aligned at 4 bytes
for(#types)lengthshort2Length in bytes of type (e.g. 4)
structure identifierchar[4]4'STRC' this field is aligned at 4 bytes
#structuresinteger4Number of structures follows
for(#structures)structure typeshort2Index in types containing the name of the structure
..#fieldsshort2Number of fields in this structure
..for(#field)field typeshort2Index in type
for endfor endfield nameshort2Index in name
As you can see, the structures are stored in 4 arrays: names, types, lengths and structures. Every structure also contains an array of fields. A field is the combination of a type and a name. From this information a catalog of all structures can be constructed. The names are stored as how a C-developer defines them. This means that the name also defines pointers and arrays. (When a name starts with '*' it is used as a pointer. when the name contains for example '[3]' it is used as a array of 3 long.) In the types you'll find simple-types (like: 'integer', 'char', 'float'), but also complex-types like 'Scene' and 'MetaBall'. 'TLEN' part describes the length of the types. A 'char' is 1 byte, an 'integer' is 4 bytes and a 'Scene' is 1376 bytes long.
Note
All identifiers, are arrays of 4 chars, hence they are all aligned at 4 bytes.

Example
Created with blender 2.54.0 on little-endian hardware with a 32 bits pointer length.

The names array



The first names are: *next, *prev, *data, *first, *last, x, y, xmin, xmax, ymin, ymax, *pointer, group, val, val2, type, subtype, flag, name[32], ...
file-block-data identifier='SDNA' array-id='NAME' number of names=3019 | | | 0004 B070 01 00 00 00 [53 44 4E 41][4E 41 4D 45] [CB 0B 00 00] ....SDNANAME.... 0004 B080 [2A 6E 65 78 74 00][2A 70 72 65 76 00] [2A 64 61 74 *next.*prev.*dat | | | '*next\0' '*prev\0' '*dat' .... .... (3019 names)
Note
While reading the DNA you'll will come across some strange names like '(*doit)()'. These are method pointers and Blender updates them to the correct methods.

The types array



The first types are: char, uchar, short, ushort, int, long, ulong, float, double, void, Link, LinkData, ListBase, vec2s, vec2f, ...
array-id='TYPE' | 0005 2440 6F 6C 64 5B 34 5D 5B 34 5D 00 00 00 [54 59 50 45] old[4][4]...TYPE 0005 2450 [C9 01 00 00] [63 68 61 72 00] [75 63 68 61 72 00][73 ....char.uchar.s | | | | number of types=457 'char\0' 'uchar\0' 's' .... .... (457 types)

The lengths array


char uchar ushort short array-id length length length length 'TLEN' 1 1 2 2 0005 3AA0 45 00 00 00 [54 4C 45 4E] [01 00] [01 00] [02 00] [02 00] E...TLEN........ .... 0005 3AC0 [08 00] [04 00] [08 00] [10 00] [10 00] [14 00] [4C 00] [34 00] ............L.4. 8 4 8 ListBase vec2s vec2f ... etc length len length .... .... (457 lengths, same as number of types)

The structures array


array-id='STRC' | 0005 3E30 40 00 38 00 60 00 00 00 00 00 00 00 [53 54 52 43] @.8.`.......STRC 0005 3E40 [8E 01 00 00] [0A 00] [02 00] [0A 00] [00 00] [0A 00] [01 00] ................ 398 10 2 10 0 10 0 number of index fields index index index index structures in types in types in names in types in names ' '----------------' '-----------------' ' ' field 0 field 1 ' '--------------------------------------------------------' structure 0 .... .... (398 structures, each one describeing own type, and type/name for each field)
The DNA structures inside a Blender 2.48 blend-file can be found at http://www.atmind.nl/blender/blender-sdna.html. If we understand the DNA part of the file it is now possible to read information from other parts file-blocks. The next section will tell us how.

Reading scene information


  • the file-block identifier is 'SC'+0x00h
  • the SDNA index is 150
  • the file-block size is 1404 bytes
Now note that:
  • the structure at index 150 in the DNA is a structure of type 'Scene' (counting from 0).
  • the associated type ('Scene') in the DNA has the length of 1404 bytes.

We can map the Scene structure on the data of the file-blocks. But before we can do that, we have to flatten the Scene-structure.
struct Scene { ID id; // 52 bytes long (ID is different a structure) AnimData *adt; // 4 bytes long (pointer to an AnimData structure) Object *camera; // 4 bytes long (pointer to an Object structure) World *world; // 4 bytes long (pointer to an Object structure) ... float cursor[3]; // 12 bytes long (array of 3 floats) ... };
The first field in the Scene-structure is of type 'ID' with the name 'id'. Inside the list of DNA structures there is a structure defined for type 'ID' (structure index 17).
struct ID { void *next, *prev; struct ID *newid; struct Library *lib; char name[24]; short us; short flag; int icon_id; IDProperty *properties; };
The first field in this structure has type 'void' and name '*next'.

Looking in the structure list there is no structure defined for type 'void': it is a simple type and therefore the data should be read. The name '*next' describes a pointer. As we see, the first 4 bytes of the data can be mapped to 'id.next'.
Using this method we'll map a structure to its data. If we want to read a specific field we know at which offset in the data it is located and how much space it takes.
The next table shows the output of this flattening process for some parts of the Scene-structure. Not all rows are described in the table as there is a lot of information in a Scene-structure.
Flattened SDNA structure 150: Scene
referencestructuretypenameoffsetsizedescription
id.nextIDvoid*next04Refers to the next scene
id.prevIDvoid*prev44Refers to the previous scene
id.newidIDID*newid84
id.libIDLibrary*lib124
id.nameIDcharname[24]1624'SC'+the name of the scene as displayed in Blender
id.usIDshortus402
id.flagIDshortflag422
id.icon_idIDinticon_id444
id.propertiesIDIDProperty*properties484
adtSceneAnimData*adt524
cameraSceneObject*camera564Pointer to the current camera
worldSceneWorld*world604Pointer to the current world
Skipped rows
r.xschRenderDatashortxsch3822X-resolution of the output when rendered at 100%
r.yschRenderDatashortysch3842Y-resolution of the output when rendered at 100%
r.xpartsRenderDatashortxparts3862Number of x-part used by the renderer
r.ypartsRenderDatashortyparts3882Number of x-part used by the renderer
Skipped rows
gpdScenebGPdata*gpd13764
physics_settings.gravityPhysicsSettingsfloatgravity[3]138012
physics_settings.flagPhysicsSettingsintflag13924
physics_settings.quick_cache_stepPhysicsSettingsintquick_cache_step13964
physics_settings.rtPhysicsSettingsintrt14004
We can now read the X and Y resolution of the Scene:
  • the X-resolution is located on offset 382 of the file-block-data and must be read as a short.
  • the Y-resolution is located on offset 384 and is also a short
Note
An array of chars can mean 2 things. The field contains readable text or it contains an array of flags (not humanly readable).
Note
A file-block containing a list refers to the DNA structure and has a count larger than 1. For example Vertexes and Faces are stored in this way.