The best idea inside rigid body simulation is that, you can use one single point and its state ( orientation & movement ) to represent the whole object. I used Houdini to dig further into this feature. I did a Destruction Simulation.
The tutorial I used is: https://www.sidefx.com/tutorials/appliedrigidbodydestruction/
and my result looks like this:
Because these are just choruses instead of some cool model, some of the effort in there can not be told without observation. Now I will explain step by step (not digging into the houdini nodes) how to make this simulation looks real.

Pack Object
It is very natural to think of packing an object. Packing means store all the mesh information in the memory and use a point to represent its state and reference to that mesh. Packing can make preview runs faster but it doesn’t save real rendering time.
2. Give It a Bullet Solver
Here is the link to explain bullet solver in houdini: http://www.sidefx.com/ja/docs/houdini/nodes/dop/bulletrbdsolver
In a word, the bullet solver can represent the object with simpler shape. It can use convex shapes with less points to represent the object in any situation, which is especially useful in collision detection.
3. Voronoi: Divide the object in space
Voronoi is very fast and the pieces it devided into are very simple. Because of that, the simulation looks a little fake when you look closely. With fogs and motion blur it might look ok.
So the way to do this is to give voronoi the original mesh and some control points around the mesh. The points on the mesh will be clustered with the nearest control point. That is, the mesh is divided by bisections of the control points.
By now, if you give the object a height and drop it onto the ground, it will be broken into pieces. However, if you give it a initial angular velocity, it will be Broken Immediately. The reason is that, they are just pieces that start with a position with the same velocity. They are not bounded together.
Also, the pieces it generated are too regular. To do that, you can use voronoi’s own clustering. But it is not good at all. It can create interpenetrating problem between pieces so, not recommended.
4. Add constraints (glue constraints in houdini)
To bound them together you can create connections that linked adjacent pieces together after you use Voronoi. With appropriate search radius and max connections, you can create beautiful link network. I deleted the connections that exist outside the mesh. After that, you can give the connections strengths. In Houdini you can write expressions to give connections random chance to be extremely strong.
After that, you input the constraints inside the simulation. It improves both problems: the pieces will not flew apart with an angular velocity and also, the part that falls on the ground will break the most, and the others will generate bigger parts because of the constraints’ strength.
5. Break it again with smaller pieces
By now we have regular pieces and big pieces, but it should have more small pieces that fall apart. To do this, the tutorial breaks random pieces again before it is assembled into another pack.
The result looks like this:
It looks quite nice with all the details. The breaking time looks reasonable, and the pieces generated contain all kinds of shapes. It will look nicer with dust and smoke.