Cocoa Graphics Frameworks
I've had some people ask how all the different graphics frameworks in Mac OS X fit together from the perspective of a Cocoa programmer. I'm not quite the expert on all of this, but I think I can help clear up a few things.(thanks to Josh Anon for help sorting out ImageIO and Quartz Composer)
Quartz and Core Graphics
Quartz is the basis for virtually all Mac OS X graphics and the window server. The Quartz drawing API is exposed in CoreGraphics.framework. It's a procedural C-based framework, but it uses the Core Foundation memory management rules so things at least feel close to Objective-C's reference counting system.
You can also use Quartz functionality in Cocoa with classes like NSBezierPath, NSImage and NSGraphicsContext. They're easier to use and somewhat interchangeable with Core Graphics data structures.
The drawing model in Quartz is based on PDF. That doesn't mean everything is a PDF, it means that a lot of the concepts are borrowed from PDF. For example, Cocoa views start their coordinate system in the lower-left of a rectangle by default.
All Quartz coordinates are in floating-point values. There's no assumption that a value of 1.0 equals one pixel on the screen, which means that the same drawing code can be used for devices at different resolutions.
Quartz Extreme, which arrived in Mac OS X 10.2 Jaguar, can combine 2D graphics, 3D graphics and video into a single stream and send the result to the GPU for final processing using OpenGL. The result is that you can have fancy 3D graphics overlaid on DVD video, video in 3D environments, and so on -- all hardware accelerated (this was four years ago, dude!).
Quartz Extreme compositing and acceleration happens automatically, so it's not something you have to write any code to use.
Some key Core Graphics data structures are CGGraphicsContext, CGImage, CGPoint, CGPath, CGRect, CGFont, and CGColor.
The Quartz2DBasics sample code from Apple is a pretty good starting point for using Core Graphics, and of course there's the Quartz topic page too.
OpenGL
On Mac OS X, OpenGL is generally used in games, mathematic and scientific visualization, and 3D design apps. However, Quartz Extreme uses OpenGL to accelerate compositing and drawing. So in a sense, all Mac OS X apps indirectly use OpenGL.
OpenGL can be used in CoreGraphics with the data structures in OpenGL.framework, and CoreVideo has CVOpenGLBuffer and CVOpenGLTexture.
Apple OpenGL topic page is first step, and the there's a good selection of sample code.
Core Image and Core Video
Core Image and Core Video can manipulate image and video streams with real-time effects. You can see this sort of thing in action in Final Cut Pro, Motion and Aperture. You need relatively new hardware (2005-ish) to do real-time effects. Core Image is an Objective-C based API, but Core Video is a procedural C-based API, in the spirit of Core Foundation.
Apple has a nice walkthrough of Core Image.
ImageIO and Image Capture
ImageIO (technically, Image I/O) is what you want to use to get image data in and out of Quartz. Cocoa's Objective-C image classes can load files from disk too, but ImageIO gives you the ability to fetch only the image headers, which can result in much faster loading in a image viewing app.
ImageIO can also extract image metadata, read and write thumbnails, and pull apart files that contain, say, both jpeg and raw versions of the same image.
The Image Capture framework helps you grab images from cameras and scanners. It's a lower-level procedural C-based API, but it does the job.
QuickTime and QTKit
QuickTime is Apple's swiss army knife for handling motion media, most importantly video. The catch for Cocoa programmers is that it was designed in the early days of the Mac, so the raw C-based API looks like old Mac Toolbox code in some parts.
Starting in Mac OS X 10.4 Tiger, though, Cocoa programmers can skip a lot of that and just use QTKit instead. QTKit is an Objective-C framework for QuickTime. QTKit doesn't yet have all of the same features as the lower-level raw C API, but it provides a good chunk of the most important bits.
Just to be clear, QuickTime is what you use for the bulk of displaying and editing of video. Core Video is used to apply real-time affects to video.
QTKit is described nicely in a feature article from Apple, and QuickTime has its own topic page.
Quartz Composer
There are two parts to Quartz Composer: the framework and the application. The framework allows you to bundle different kinds of Mac OS X graphics content into compositions. The compositions can be easily used in Cocoa applications with the QCView class.
The real fun, though, is the Quartz Composer application, which lets you explore all these fancy tools visually. The results can be saved out and manipulated in code. Quartz Composer can produce some amazing results, but in all fairness, it's quite complicated.
Quartz Composer is profiled in a feature article from Apple.
Core Animation
Core Animation is a new framework coming in Mac OS X 10.5 Leopard. Most of it is still under NDA, but here's the main idea: you create layers which can contain virtually any Mac OS X graphics content: 2D graphics, video, OpenGL, and so on. You can then simply adjust layer properties and Core Animation works out all animation details.
The basic idea here is to make custom interfaces much easier to create. The demos Apple has shown revolve around fancy 3D animations, but there's nothing that says you have to make things 3D or even animated. For example, Xcode 3 uses Core Animation for a number of new UI elements.
It looks like there is some overlap with Quartz Composer, but the thing to focus on here is that Core Animation is designed to be very easy to use and produce stellar results. Core Animation is also smart about using threads to offload work to idle processor cores.
In short, they really nailed this one. This is the most interesting new API I've seen for Leopard so far. Apple has a preview page for Core Animation up on the Leopard site.
PDFKit
PDFKit is a high-level Objective-C framework for reading, displaying and writing PDF documents. If you have a use for it, it's a true gift. Instead of going on for paragraphs about this, I'm just going to point you at Apple's feature article on PDFKit.
Finding the Frameworks
So once you know which framework you want to use, you need to figure where the actual framework file is that you want to use. They're not all in the most obvious places, and some of the names are a bit confusing, so here's a rough map.
All system-level frameworks live somewhere under /System/Library/Frameworks, but some are bundled in umbrella frameworks like ApplicationServices.
Common Name Framework
Quartz ApplicationServices/CoreGraphics.framework
OpenGL OpenGL.framework
CoreImage QuartzCore.framework
CoreVideo QuartzCore.framework
ImageIO ApplicationServices/ImageIO.framework
ImageCapture Carbon/ImageCapture.framework
QuickTime QuickTime.framework
QTKit QTKit.framework
QuartzComposer Quartz/QuartzComposer.framework
CoreAnimation (not out yet! 10.5 Leopard)
PDFKit Quartz/PDFKit.frameworkOne last note. You might want to take a look at Apple's Getting Started with Graphics and Imaging page.
Cocoa Graphics Frameworks
Posted Sep 22, 2006 — 18 comments below
Posted Sep 22, 2006 — 18 comments below
Jeroen Leenarts — Sep 22, 06 1839
Core Graphics uses a unit called a point and one point is defined as 1/72 th of an inch. So drawing something at a length of 72 points should yield roughly a 1 inch stretch on screen, no matter what resolution your screen is at.
Do note I haven't tested this yet. But several sources seem to support this claim. Also an excellent read on Quartz 3D/Core Graphics: ISBN 0-321-33663-1 It's Quarzt 2D Graphics for Mac OS X Developers by R.S. Thompson.
Joachim Bengtsson — Sep 22, 06 1840
I have to say, PDFKit is pretty awesome. I've been using it for an internal application, and the ability to both use a PDFView which requires *no code*, but still use PDFPage and ask it to render itself to my own view if I have to, is all ridiculously powerful easy. Throw a mail my way if you have any questions about it.
By the way, NSAffineTransformations are cool, will make any OpenGL nerd gleeful, and are really important to understand if you're to make interesting custom views. At least before we get Core Animation...
Jean-Franois Roy — Sep 22, 06 1841
I think that's wrong now. Quartz and OpenGL are arguably as important as the other in modern Mac OS X. Quartz relies on OpenGL to composite and draw on screen, OpenGL relies on Quartz for high-quality rasterizing of vector graphics. OpenGL is used more and more in plain application UIs, such as the cover art view in iTunes 7. What makes Mac OS X great is the synergy between those two.
A few other points:
• Core Animation isn't a graphics library more than it is a layering / compositing / animation framework. OpenGL and Quartz generate images or graphic objects, Core Animation manipulates and arranges in space and time.
• Core Video truly is a buffer management and timing framework aimed at video sequences. It provides numerous types of buffers and buffer pools generally differentiated by the backing store of the buffers (main memory, GPU VRAM or OpenGL texture in GPU VRAM). It's main usage is real-time video processing because it's been integrated into QuickTime's decoding and encoding APIs. It could be used for any kind of buffer management problems. such as GPUGP.
• Mac OS X is awesome :)
Uli Kusterer — Sep 22, 06 1842
That hasn't been true in ages for the screen, though it is true for printing, unless you've it set up to scale, of course.
MacOS X currently assumes a screen resolution of 72dpi, but most screens you buy today are actually more like 80 or 90dpi. So, in general stuff displays too small on screen.
That's why resolution independence has been introduced: To allow you to use the 72dpi units and the user can choose whether they want to see more content or get higher resolution. Right now you always get more resolution, in the future you can scale up and thus get 1:1 again, but with the same smoothness for text and graphics.
At least that's the theory.
Uli Kusterer — Sep 22, 06 1843
Scott Stevenson — Sep 22, 06 1844
That's just at the lower levels, though. Cocoa programmers are more likely to write Quartz code than OpenGL code.
The compositor does use OpenGL to get data onto the GPU. That's not something you have to really be aware of at the Cocoa level, though. It could change at any time without breaking existing Quartz apps.
Qwerty Denzel — Sep 22, 06 1845
Now avoids use of floats where unnecessary. Where floating point constants are necessary avoids explicit use (now uses 10.0 instead of 10.0f). These changes make the code more readable while preserving efficiency and make the code ready for conversion to 64 bit.
I've mostly used the form without the suffix, simply because I did find it more readable, but is this something official that I need the adhere to?
It also says the change makes ready for conversion to 64 bit. What difference does it make?
Jean-Franois Roy — Sep 22, 06 1846
It also says the change makes ready for conversion to 64 bit. What difference does it make?"
It's a simple matter of precision. By using x.y, the compiler will generate doubles, which will be implicitly converted to floats if assigned to float variables, such as used in Quartz in 32 bits program. When you flick the 64 bits switch, you'll magically get double the precision you had in your draw code.
By using x.yf, the compiler will generate a float, and if you then assign that float to a double variable, the compiler will also generate an implicit type conversion and further math using that double variable will benefit from the added precision.
However, if in your various classes and algorithms you have float member variables or local variables, then you will not get those benefits, and even risk accentuating rounding errors.
Qwerty Denzel — Sep 22, 06 1847
It also says the change makes it ready for conversion to 64 bit.
Also, I think it needs to be mentioned that Quartz Extreme requires at least 16MB of Video-RAM, and so will not work on some of Tiger's oldest supported computers (like my Pismo, which, from what I been reading, looks like it won't be supported in Leopard).
Qwerty Denzel — Sep 22, 06 1848
Preston — Sep 23, 06 1849
"Jeroen Leenarts: I think that with 10.5, you can't make that assumption anymore, as the UI is scaleable. I'm not sure, though. If you want to make pixel-exact drawing, you could always read more about Pixel exact drawing in the documentation ;)"
Actually, one of the reasons for resolution independence is to allow you to make length assumptions regardless of the device's output resolution. In the new system, you target points as before, but now those points can automaticaly scale to 72 dpi or whatever the user is using on their device. I think what you meant to say was that without 10.5, you can't make those assumptions because of non-constant device dpi.
Boyan — Sep 24, 06 1850
I'm not much into MacOSX frameworks, but for any graphic programming you should remember that GPUs basically operate on 32-bit floats (any vertex/pixel shader unit works on vectors - 4x32-float values). Any cast performed by CPU to bring doubles to 32-bit floats is just wasted CPU time.
Of course, I'm not saying don't use doubles at all, I'm just not sure where double format fits in graphics-framework pipeline. Definitely not anywhere near low-level code
Steve Israelson — Sep 26, 06 1852
One is still vapour.
Res independence is not coming in Leopard. Apple clearly stated 2008, so maybe a point release?
Philip Orr — Oct 12, 06 2045
Phil
Scott Stevenson — Oct 12, 06 2049
Core Animation is a framework, so it's part of the OS (specifically, Leopard), not Xcode.
how do developers create CoreAnimation applications to be released with 10.5
You sign up for an ADC membership which gives you access to Leopard seeds.
Darren — Oct 29, 06 2214
The Leopard Technology Overview does not mention 2008. It simply gives an explanation of the subject under the heading of Resolution Independence. So I'm expected it in with the first release of Leopard.
Abayomi — Mar 05, 09 6616
vijay — Sep 15, 09 6866
i need to know some links form which i can create iphone screen savers........