Cocoa Graphics FrameworksI've had some people ask how all the different graphics frameworks in Mac OS X fit together from the perspective of a Cocoa programmer. I'm not quite the expert on all of this, but I think I can help clear up a few things.
(thanks to Josh Anon for help sorting out ImageIO and Quartz Composer)
Quartz and Core Graphics
Quartz is the basis for virtually all Mac OS X graphics and the window server. The Quartz drawing API is exposed in CoreGraphics.framework. It's a procedural C-based framework, but it uses the Core Foundation memory management rules so things at least feel close to Objective-C's reference counting system.
You can also use Quartz functionality in Cocoa with classes like NSBezierPath, NSImage and NSGraphicsContext. They're easier to use and somewhat interchangeable with Core Graphics data structures.
The drawing model in Quartz is based on PDF. That doesn't mean everything is a PDF, it means that a lot of the concepts are borrowed from PDF. For example, Cocoa views start their coordinate system in the lower-left of a rectangle by default.
All Quartz coordinates are in floating-point values. There's no assumption that a value of 1.0 equals one pixel on the screen, which means that the same drawing code can be used for devices at different resolutions.
Quartz Extreme, which arrived in Mac OS X 10.2 Jaguar, can combine 2D graphics, 3D graphics and video into a single stream and send the result to the GPU for final processing using OpenGL. The result is that you can have fancy 3D graphics overlaid on DVD video, video in 3D environments, and so on -- all hardware accelerated (this was four years ago, dude!).
Quartz Extreme compositing and acceleration happens automatically, so it's not something you have to write any code to use.
Some key Core Graphics data structures are CGGraphicsContext, CGImage, CGPoint, CGPath, CGRect, CGFont, and CGColor.
The Quartz2DBasics sample code from Apple is a pretty good starting point for using Core Graphics, and of course there's the Quartz topic page too.
On Mac OS X, OpenGL is generally used in games, mathematic and scientific visualization, and 3D design apps. However, Quartz Extreme uses OpenGL to accelerate compositing and drawing. So in a sense, all Mac OS X apps indirectly use OpenGL.
OpenGL can be used in CoreGraphics with the data structures in OpenGL.framework, and CoreVideo has CVOpenGLBuffer and CVOpenGLTexture.
Apple OpenGL topic page is first step, and the there's a good selection of sample code.
Core Image and Core Video
Core Image and Core Video can manipulate image and video streams with real-time effects. You can see this sort of thing in action in Final Cut Pro, Motion and Aperture. You need relatively new hardware (2005-ish) to do real-time effects. Core Image is an Objective-C based API, but Core Video is a procedural C-based API, in the spirit of Core Foundation.
Apple has a nice walkthrough of Core Image.
ImageIO and Image Capture
ImageIO (technically, Image I/O) is what you want to use to get image data in and out of Quartz. Cocoa's Objective-C image classes can load files from disk too, but ImageIO gives you the ability to fetch only the image headers, which can result in much faster loading in a image viewing app.
ImageIO can also extract image metadata, read and write thumbnails, and pull apart files that contain, say, both jpeg and raw versions of the same image.
The Image Capture framework helps you grab images from cameras and scanners. It's a lower-level procedural C-based API, but it does the job.
QuickTime and QTKit
QuickTime is Apple's swiss army knife for handling motion media, most importantly video. The catch for Cocoa programmers is that it was designed in the early days of the Mac, so the raw C-based API looks like old Mac Toolbox code in some parts.
Starting in Mac OS X 10.4 Tiger, though, Cocoa programmers can skip a lot of that and just use QTKit instead. QTKit is an Objective-C framework for QuickTime. QTKit doesn't yet have all of the same features as the lower-level raw C API, but it provides a good chunk of the most important bits.
Just to be clear, QuickTime is what you use for the bulk of displaying and editing of video. Core Video is used to apply real-time affects to video.
QTKit is described nicely in a feature article from Apple, and QuickTime has its own topic page.
There are two parts to Quartz Composer: the framework and the application. The framework allows you to bundle different kinds of Mac OS X graphics content into compositions. The compositions can be easily used in Cocoa applications with the QCView class.
The real fun, though, is the Quartz Composer application, which lets you explore all these fancy tools visually. The results can be saved out and manipulated in code. Quartz Composer can produce some amazing results, but in all fairness, it's quite complicated.
Quartz Composer is profiled in a feature article from Apple.
Core Animation is a new framework coming in Mac OS X 10.5 Leopard. Most of it is still under NDA, but here's the main idea: you create layers which can contain virtually any Mac OS X graphics content: 2D graphics, video, OpenGL, and so on. You can then simply adjust layer properties and Core Animation works out all animation details.
The basic idea here is to make custom interfaces much easier to create. The demos Apple has shown revolve around fancy 3D animations, but there's nothing that says you have to make things 3D or even animated. For example, Xcode 3 uses Core Animation for a number of new UI elements.
It looks like there is some overlap with Quartz Composer, but the thing to focus on here is that Core Animation is designed to be very easy to use and produce stellar results. Core Animation is also smart about using threads to offload work to idle processor cores.
In short, they really nailed this one. This is the most interesting new API I've seen for Leopard so far. Apple has a preview page for Core Animation up on the Leopard site.
PDFKit is a high-level Objective-C framework for reading, displaying and writing PDF documents. If you have a use for it, it's a true gift. Instead of going on for paragraphs about this, I'm just going to point you at Apple's feature article on PDFKit.
Finding the Frameworks
So once you know which framework you want to use, you need to figure where the actual framework file is that you want to use. They're not all in the most obvious places, and some of the names are a bit confusing, so here's a rough map.
All system-level frameworks live somewhere under /System/Library/Frameworks, but some are bundled in umbrella frameworks like ApplicationServices.
Common Name Framework
CoreAnimation (not out yet! 10.5 Leopard)
One last note. You might want to take a look at Apple's Getting Started with Graphics and Imaging page.
Cocoa Graphics Frameworks
Posted Sep 22, 2006 — 18 comments below
Posted Sep 22, 2006 — 18 comments below